Taskonomy: Disentangling Task Transfer Learning

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Taskonomy: Disentangling Task Transfer Learning".

Jane: Here's the gist of the paper. The gist The authors propose a fully computational approach for modeling the structure of visual tasks by finding transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space to create a computational taxonomic map for task transfer learning<ref:1804.08328#pg2> Modeling the Task Space The paper proposes finding (first and higher-order) transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space to create a computational taxonomic map for task transfer learning<ref:1804.08328#pg2> This structure is defined as "a collection of computationally found relations specifying which tasks supply useful information to another, and by how much" <ref:1804.08328#pg3> The approach employs a fully computational method with neural networks as the adopted computational function class<ref:1804.08328#pg3> In a feedforward network, each layer successively forms more abstract representations of the input containing the information needed for mapping the input to the output<ref:1804.08328#pg4> The basis of our approach is that we compute an affinity matrix among tasks based on whether the solution for one task can be sufficiently easily read out of the representation trained for another task<ref:1804.08328#pg5> The Four-Step Taxonomy Process Taxonomy is built using a four step process depicted in Fig.

Tom: 2<ref:1804.08328#pg6> In stage I, a task-specific network for each task in S Query Image is trained<ref:1804.08328#pg7> In stage II, all feasible transfers between sources and targets are trained Higher-Order Transfers: Multiple source tasks can contain complementary information for solving a target task In stage III, the task affinities acquired from transfer function performances are normalized, and in stage IV, we synthesize a hypergraph which can predict the performance of any transfer policy and optimize for the optimal one Transfer Modeling Details In step II, given a source task s and a target task t, where s ∈ S and t ∈ T, a transfer network learns a small readout function for t given a statistic computed for s The statistic is the representation for image I from the encoder of s: Es(I) The readout function (Ds→t) is parameterized by θs→t minimizing the loss Lt: Ds→t:= arg min θ EI∈Dh ft(I) i, (1) Accessibility: For a transfer to be successful, the latent representation of the source should both be inclusive of sufficient information for solving the target and have the information accessible We use a shallow fully convolutional network and train it with little data (8x to 120x less than task-specific networks) in order to measure transferability conditioned on being highly accessible Higher-Order Transfers: Multiple source tasks can contain complementary information for solving a target task We include higher-order transfers which are the same as first order but receive multiple representations in the input Ordinal Normalization and Global Policy In step III, we use an ordinal approach in which the output quality and loss are only assumed to change monotonically For each t, we construct Wt a pairwise tournament matrix between all feasible sources for transferring to t The final tournament ratio matrix is positive reciprocal with each element w 0 i,j of W 0t We stack the principal eigenvectors of W 0t for all t ∈ T, to get an affinity matrix P (‘p’ for performance) In step IV, we devise a global transfer policy which maximizes collective performance across all tasks, while minimizing the used supervision This problem is formulated as subgraph selection where tasks are nodes and transfers are edges We solve this subgraph selection problem using Boolean Integer Programming (BIP), described below, which can be solved optimally and efficiently Experimental Results and Significance The experiments show that the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3 (compared to training independently) while keeping the performance nearly the same The computed taxonomies optimized to solve the full dictionary, i.e.

Jane: all tasks are placed in T and S (except for 4 source-only tasks that are in S only) The taxonomy outperformed all other connectivities by a large margin signifying both existence of a strong structure in the space as well as a good modeling of it The results show that the space has a rather predictable and strong structure The Spearman’s rho between the taxonomy ranking and the Top-1 ranking is 0.857 on Places and 0.823 on ImageNet showing a notable correlation We found that the closer two tasks, the more similar their role in transferring to other tasks The 3D, 2D, low dimensional geometric, and semantic tasks are found to cluster together using a fully computational approach Limitations Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class Compositionality: We performed the modeling via a set of common human-defined visual tasks Space Regularity: We performed modeling of a dense space via a sampled dictionary Model Dependence: We used a computational approach and adopted neural networks as our function class<ref:1804.

Title and authors: Tom: Let's talk about the title and the authors of Taskonomy: Disentangling Task Transfer Learning. What do those names tell us?

Jane: The title really gets to the core of what they’re trying to do—disentangle that structure among different visual tasks, which is key for transfer learning.

Lu: The authors are using a fully computational approach with neural networks as their function class to model this space <ref:1804.08328#pg2>.

Meng: It’s interesting they are using this setup to find these dependencies across twenty six different visual tasks, which is a pretty large dictionary to work with.

Lalam: This whole mapping process is designed so that the resulting structure can be used as a principled way to identify redundancies across tasks <ref:1804.08328#pg2>.

The paper's summary: Tom: So, what’s the summary of Taskonomy? How do they actually model this task space?

Jane: They build a four-step taxonomy process that starts with training task-specific networks for each individual task in the set <ref:1804.08328#pg6>.

Lu: Step two involves training transfer functions between sources and targets, where they learn a small readout function for one task given a statistic computed from another source, like Es(I) <ref:1804.08328#pg5>.

Meng: They are finding these first-order and higher-order transfers, meaning multiple source tasks can provide complementary information for solving a target task <ref:1804.08328#pg6>.

Lalam: Then in step three, they normalize those task affinities using an Analytic Hierarchy Process to get a pairwise tournament matrix, and finally, in step four, they synthesize a hypergraph that predicts the performance of any transfer policy <ref:1804.08328#pg6>.

The paper's improvements: Tom: So what are the actual improvements they propose with this taxonomy? What does it actually do better than just training things separately?

Jane: The main improvement is that by computing this taxonomy, you can get provably efficient comprehensive perception models <ref:1804.08328#pg2>. They aim for models that can solve a large set of tasks before they become too demanding in terms of supervision or computation <ref:1804.08328#pg2>.

Lu: The paper shows that by following this taxonomy, you can reduce the total number of labeled data points needed to solve a set of ten tasks by roughly two-thirds compared to training them independently while keeping performance almost the same <ref:1804.08328#pg7>.

Meng: That’s significant because it directly addresses the cost and complexity of training models for many different things at once, which is something I deal with every day when deploying models.

Lalam: They also find that the taxonomy they compute actually outperformed all other connectivities by a large margin, showing a strong structure in the space <ref:1804.08328#pg7>.

Conclusion: Tom: Alright, to wrap this up, what's the final word on Taskonomy and what does it mean for how we think about visual AI?

Jane: It suggests that the space of visual tasks has a rather predictable and strong structure, and that tasks that are closer together in this taxonomy have a more similar role in transferring to other tasks <ref:1804.08328#pg7>.

Lu: The clustering they found—where three dee, 2D, low dimensional geometric, and semantic tasks all group together—gives us a way to understand how these visual concepts are related hierarchically <ref:1804.08328#pg7>.

Meng: For deployment at scale, this structure means we can move away from training every model in isolation and toward a system that exploits these known relationships for much better efficiency.

Lalam: Taskonomy is the computational taxonomic map they propose to map the space of visual tasks by finding transfer learning dependencies across a dictionary of twenty six 2D, 2 point 5D, three dee, and semantic tasks <ref:1804.08328#pg2>.