Taskonomy: Disentangling Task Transfer Learning

arXiv:1804.08328 · cs.CV, cs.AI, cs.LG, cs.NE, cs.RO · Submitted 2018-04-23 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Taskonomy: Disentangling Task Transfer Learning".

Jane: Here's the gist of the paper. The gist The authors propose a fully computational approach for modeling the structure of visual tasks by finding transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space to create a computational taxonomic map for task transfer learning<ref:1804.08328#pg2> Modeling the Task Space The paper proposes finding (first and higher-order) transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space to create a computational taxonomic map for task transfer learning<ref:1804.08328#pg2> This structure is defined as "a collection of computationally found relations specifying which tasks supply useful information to another, and by how much" <ref:1804.08328#pg3> The approach employs a fully computational method with neural networks as the adopted computational function class<ref:1804.08328#pg3> In a feedforward network, each layer successively forms more abstract representations of the input containing the information needed for mapping the input to the output<ref:1804.08328#pg4> The basis of our approach is that we compute an affinity matrix among tasks based on whether the solution for one task can be sufficiently easily read out of the representation trained for another task<ref:1804.08328#pg5> The Four-Step Taxonomy Process Taxonomy is built using a four step process depicted in Fig.

Tom: 2<ref:1804.08328#pg6> In stage I, a task-specific network for each task in S Query Image is trained<ref:1804.08328#pg7> In stage II, all feasible transfers between sources and targets are trained Higher-Order Transfers: Multiple source tasks can contain complementary information for solving a target task In stage III, the task affinities acquired from transfer function performances are normalized, and in stage IV, we synthesize a hypergraph which can predict the performance of any transfer policy and optimize for the optimal one Transfer Modeling Details In step II, given a source task s and a target task t, where s ∈ S and t ∈ T, a transfer network learns a small readout function for t given a statistic computed for s The statistic is the representation for image I from the encoder of s: Es(I) The readout function (Ds→t) is parameterized by θs→t minimizing the loss Lt: Ds→t:= arg min θ EI∈Dh ft(I) i, (1) Accessibility: For a transfer to be successful, the latent representation of the source should both be inclusive of sufficient information for solving the target and have the information accessible We use a shallow fully convolutional network and train it with little data (8x to 120x less than task-specific networks) in order to measure transferability conditioned on being highly accessible Higher-Order Transfers: Multiple source tasks can contain complementary information for solving a target task We include higher-order transfers which are the same as first order but receive multiple representations in the input Ordinal Normalization and Global Policy In step III, we use an ordinal approach in which the output quality and loss are only assumed to change monotonically For each t, we construct Wt a pairwise tournament matrix between all feasible sources for transferring to t The final tournament ratio matrix is positive reciprocal with each element w 0 i,j of W 0t We stack the principal eigenvectors of W 0t for all t ∈ T, to get an affinity matrix P (‘p’ for performance) In step IV, we devise a global transfer policy which maximizes collective performance across all tasks, while minimizing the used supervision This problem is formulated as subgraph selection where tasks are nodes and transfers are edges We solve this subgraph selection problem using Boolean Integer Programming (BIP), described below, which can be solved optimally and efficiently Experimental Results and Significance The experiments show that the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3 (compared to training independently) while keeping the performance nearly the same The computed taxonomies optimized to solve the full dictionary, i.e.

Jane: all tasks are placed in T and S (except for 4 source-only tasks that are in S only) The taxonomy outperformed all other connectivities by a large margin signifying both existence of a strong structure in the space as well as a good modeling of it The results show that the space has a rather predictable and strong structure The Spearman’s rho between the taxonomy ranking and the Top-1 ranking is 0.857 on Places and 0.823 on ImageNet showing a notable correlation We found that the closer two tasks, the more similar their role in transferring to other tasks The 3D, 2D, low dimensional geometric, and semantic tasks are found to cluster together using a fully computational approach Limitations Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class<ref:1804.

Title and authors: Tom: Let's talk about the title and the authors of Taskonomy: Disentangling Task Transfer Learning. What do those names tell us?

Jane: The title really gets to the core of what they’re trying to do—disentangle that structure among different visual tasks, which is key for transfer learning.

Lu: The authors are using a fully computational approach with neural networks as their function class to model this space <ref:1804.08328#pg2>.

Meng: It’s interesting they are using this setup to find these dependencies across twenty six different visual tasks, which is a pretty large dictionary to work with.

Lalam: This whole mapping process is designed so that the resulting structure can be used as a principled way to identify redundancies across tasks <ref:1804.08328#pg2>.

The paper's summary: Tom: So, what’s the summary of Taskonomy? How do they actually model this task space?

Jane: They build a four-step taxonomy process that starts with training task-specific networks for each individual task in the set <ref:1804.08328#pg6>.

Lu: Step two involves training transfer functions between sources and targets, where they learn a small readout function for one task given a statistic computed from another source, like Es(I) <ref:1804.08328#pg5>.

Meng: They are finding these first-order and higher-order transfers, meaning multiple source tasks can provide complementary information for solving a target task <ref:1804.08328#pg6>.

Lalam: Then in step three, they normalize those task affinities using an Analytic Hierarchy Process to get a pairwise tournament matrix, and finally, in step four, they synthesize a hypergraph that predicts the performance of any transfer policy <ref:1804.08328#pg6>.

The paper's improvements: Tom: So what are the actual improvements they propose with this taxonomy? What does it actually do better than just training things separately?

Jane: The main improvement is that by computing this taxonomy, you can get provably efficient comprehensive perception models <ref:1804.08328#pg2>. They aim for models that can solve a large set of tasks before they become too demanding in terms of supervision or computation <ref:1804.08328#pg2>.

Lu: The paper shows that by following this taxonomy, you can reduce the total number of labeled data points needed to solve a set of ten tasks by roughly two-thirds compared to training them independently while keeping performance almost the same <ref:1804.08328#pg7>.

Meng: That’s significant because it directly addresses the cost and complexity of training models for many different things at once, which is something I deal with every day when deploying models.

Lalam: They also find that the taxonomy they compute actually outperformed all other connectivities by a large margin, showing a strong structure in the space <ref:1804.08328#pg7>.

Conclusion: Tom: Alright, to wrap this up, what's the final word on Taskonomy and what does it mean for how we think about visual AI?

Jane: It suggests that the space of visual tasks has a rather predictable and strong structure, and that tasks that are closer together in this taxonomy have a more similar role in transferring to other tasks <ref:1804.08328#pg7>.

Lu: The clustering they found—where three dee, 2D, low dimensional geometric, and semantic tasks all group together—gives us a way to understand how these visual concepts are related hierarchically <ref:1804.08328#pg7>.

Meng: For deployment at scale, this structure means we can move away from training every model in isolation and toward a system that exploits these known relationships for much better efficiency.

Lalam: Taskonomy is the computational taxonomic map they propose to map the space of visual tasks by finding transfer learning dependencies across a dictionary of twenty six 2D, 2 point 5D, three dee, and semantic tasks <ref:1804.08328#pg2>.

Amir R. Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, Silvio Savarese

Stanford University · University of California, Berkeley

cs.CV, cs.AI, cs.LG, cs.NE, cs.RO

Submitted: 2018-04-23

Updated: 2026-10-20

Comments: CVPR 2018 (best paper). See project website and live demos at http://taskonomy.stanford.edu/

Project page: http://taskonomy.stanford.edu

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist The authors propose a fully computational approach for modeling the structure of visual tasks by finding transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and

Key concepts

Computational Taxonomic Map: This is a structural representation of visual tasks derived computationally. It maps out which tasks provide useful information to others and the strength of that relationship, creating a 'map' that shows how different tasks are connected in terms of their transferability.
Transfer Learning Dependencies: These are the specific relationships discovered between different visual tasks within a latent space. The authors find out which source tasks can provide sufficient information for solving a target task, and by what degree, forming the basis of the taxonomic map.
Four-Step Taxonomy Process: This is the systematic method used to build the taxonomy. It involves training task-specific networks, learning feasible transfers between sources and targets, normalizing these affinities into an ordinal matrix, and finally synthesizing a global policy for optimal task performance.
Boolean Integer Programming (BIP): This mathematical technique is used in the final step to select the best set of tasks (subgraph) for a given supervision budget. It mathematically optimizes the selection of source tasks and transfers to targets to maximize overall performance while respecting constraints like the available data.

Terminology

Transfer Learning — 4× in this paper · explained in 15 episodes
Latent Space — 1× in this paper · explained in 7 episodes

Summary

The gist The authors propose a fully computational approach for modeling the structure of visual tasks by finding transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space to create a computational taxonomic map for task transfer learning<ref:1804.08328#pg2>

Modeling the Task Space

The paper proposes finding (first and higher-order) transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space to create a computational taxonomic map for task transfer learning<ref:1804.08328#pg2> This structure is defined as a collection of computationally found relations specifying which tasks supply useful information to another, and by how much <ref:1804.08328#pg3> The approach employs a fully computational method with neural networks as the adopted computational function class<ref:1804.08328#pg3> In a feedforward network, each layer successively forms more abstract representations of the input containing the information needed for mapping the input to the output<ref:1804.08328#pg4> The basis of our approach is that we compute an affinity matrix among tasks based on whether the solution for one task can be sufficiently easily read out of the representation trained for another task<ref:1804.08328#pg5>

The Four-Step Taxonomy Process

Taxonomy is built using a four step process depicted in Fig. 2<ref:1804.08328#pg6> In stage I, a task-specific network for each task in S Query Image is trained<ref:1804.08328#pg7> In stage II, all feasible transfers between sources and targets are trained Higher-Order Transfers: Multiple source tasks can contain complementary information for solving a target task In stage III, the task affinities acquired from transfer function performances are normalized, and in stage IV, we synthesize a hypergraph which can predict the performance of any transfer policy and optimize for the optimal one

Transfer Modeling Details

In step II, given a source task s and a target task t, where s ∈ S and t ∈ T, a transfer network learns a small readout function for t given a statistic computed for s The statistic is the representation for image I from the encoder of s: Es(I) The readout function (Ds→t) is parameterized by θs→t minimizing the loss Lt: Ds→t:= arg min θ EI∈Dh ft(I) i, (1) Accessibility: For a transfer to be successful, the latent representation of the source should both be inclusive of sufficient information for solving the target and have the information accessible We use a shallow fully convolutional network and train it with little data (8x to 120x less than task-specific networks) in order to measure transferability conditioned on being highly accessible Higher-Order Transfers: Multiple source tasks can contain complementary information for solving a target task We include higher-order transfers which are the same as first order but receive multiple representations in the input

Ordinal Normalization and Global Policy

In step III, we use an ordinal approach in which the output quality and loss are only assumed to change monotonically For each t, we construct Wt a pairwise tournament matrix between all feasible sources for transferring to t The final tournament ratio matrix is positive reciprocal with each element w 0 i,j of W 0t We stack the principal eigenvectors of W 0t for all t ∈ T, to get an affinity matrix P (‘p’ for performance) In step IV, we devise a global transfer policy which maximizes collective performance across all tasks, while minimizing the used supervision This problem is formulated as subgraph selection where tasks are nodes and transfers are edges We solve this subgraph selection problem using Boolean Integer Programming (BIP), described below, which can be solved optimally and efficiently

Experimental Results and Significance

The experiments show that the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3 (compared to training independently) while keeping the performance nearly the same The computed taxonomies optimized to solve the full dictionary, i.e. all tasks are placed in T and S (except for 4 source-only tasks that are in S only) The taxonomy outperformed all other connectivities by a large margin signifying both existence of a strong structure in the space as well as a good modeling of it The results show that the space has a rather predictable and strong structure The Spearman’s rho between the taxonomy ranking and the Top-1 ranking is 0.857 on Places and 0.823 on ImageNet showing a notable correlation We found that the closer two tasks, the more similar their role in transferring to other tasks The 3D, 2D, low dimensional geometric, and semantic tasks are found to cluster together using a fully computational approach

Limitations

Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class<ref:1804.

Improvements for AI systems

Improve data labeling efficiency by reducing labeled data requirements by roughly 2/3 for solving a set of 10 tasks compared to training independently while keeping performance nearly the same. This is achieved by finding transfer learning dependencies across a dictionary of visual tasks, resulting in a computational taxonomic map for task transfer learning.
Develop systems capable of solving many tasks in one system without piling up complexity by exploiting redundancies among related tasks identified through the computational taxonomy. This allows for seamlessly reuse supervision among related tasks or solve many tasks in one system without piling up the complexity.
Create a model aware of task relationships that demands less supervision, uses less computation, and behaves in more predictable ways to achieve provably efficient comprehensive/universal perception models. This moves beyond training tasks in isolation by incorporating a structure that specifies which tasks supply useful information to another, and by how much.
Implement an interactive taxonomy solver that suggests data-efficient curricula for users. This tool can be used to devise efficient supervision policies for use cases based on the discovered task space structure.
Design a system where performance is optimized by selecting the optimal subgraph from a set of tasks using Boolean Integer Programming (BIP). This BIP formulation maximizes collective performance subject to a limited supervision budget, ensuring that the optimal subgraph picks the ideal source nodes and the best edges from these sources to targets while satisfying that the number of source nodes does not exceed the supervision budget.
Enhance generalization to novel out-of-dictionary tasks by localizing where a target would be in the taxonomy using small transfer networks trained on limited data. This approach shows that carefully selecting transfer policies depending on the target is superior to fixed transfers, such as the ones employed by self-supervised methods.
Establish a similarity tree for visual tasks based on their transferring-out behavior, which clusters geometric, semantic, and 3D tasks together. This tree helps understand the how tasks would be hierarchically positioned w.r.t. to each other when measured based on providing information for solving other tasks.