Practical Skills for The AI Product Manager

In our previous article, What You Need to Know About Product Management for AI, we discussed the need for an AI Product Manager.  This role includes everything a traditional PM does, but also requires an operational understanding of machine learning software development, along with a realistic view of its capabilities and limitations.

In this article, we shift our focus to the AI Product Manager’s skill set, as it is applied to day to day work in the design, development, and maintenance of AI products. To understand the skills that product managers need, we’ll start with the process of product development, then consider how this  process differs in different kinds of organizations.

The AI Product Pipeline

We’ll start by defining the different phases of AI product development.  Though this is not an exhaustive list, most AI products pass through these stages.  In some organizations, a separate product manager shepherds the product through each stage.  Whether or not that’s how your organization works, every AI PM must consider how their products relate to these phases.  Which stage is the product in currently?  What stages will it have to go through before it becomes “real,” and how will it get there?  

Innovation/Ideation/Design for UI/X:  In traditional software engineering projects, product managers are key stakeholders in the activities that influence product and feature innovation. AI is no different. It’s incredibly important to determine what outcome is desired, how that outcome will be delivered, and how the product will be used before embarking on the long (and expensive) development journey. In the ideation phase, AI product managers should be able to use the same rapid innovation tools used by design experts, including UX mockups, wireframes, and user surveys.  At this stage, it is also critical to frame the problem or opportunity that the product addresses. In his article “Machine Learning for Product Managers,” Neal Lathia distilled ML problem types into six categories: ranking, recommendation, classification, regression, clustering, and anomaly detection. AI PMs should enter feature development and experimentation phases only after deciding what problem they want to solve as precisely as possible, and placing the problem into one of these categories. Understanding exactly what you’re doing, and how it relates to other kinds of projects, will be a huge help in researching and building solutions.

Feature Development and Data Management: This phase focuses on the inputs to a machine learning product; defining the features in the data that are relevant, and building the data pipelines that fuel the machine learning engine powering the product.

Experimentation: It’s just not possible to create a product by building, evaluating, and deploying a single model.  In reality, many candidate models (frequently hundreds or even thousands) are created during the development process.  Which model is selected for the final product is often a complex, cross-functional decision based on both qualitative and quantitative factors.  As a result, designing, implementing, and managing AI experiments (and the associated software engineering tools) is at times an AI product in itself.  Tools like MLFlow and Weights & Biases are designed to help manage experimentation.

Research: Many organizations make the mistake of hiring brilliant people with a passion for research, then putting them in a proverbial room with little to no direction and expecting “innovation” to emerge.  The result is often an overly decentralized mess that yields little value before being abandoned. The product manager for the research phase understands that AI Research products are first and foremost products, and therefore develops all of the necessary tools, structure, relationships, and resources needed to be successful. This includes product roadmaps, experiments, and investments into user interface and design. In addition, the Research PM defines and measures the lifecycle of each research product that they support.

Modelling: The model is often misconstrued as the most important component of an AI product.  In reality, the model is often the smallest amount of code in the codebase, with the smallest human dependency.  That said, repeatable success in deployment and use of a model proves elusive even for some of the most advanced organizations.  Assuming that the selected machine learning technique is suitable, the product manager will have to make several important decisions about the model. A product manager must decide whether to refactor the research code (perhaps porting it into a different language altogether), determine the scope of the ML model’s inference engine, decide on model format (for reusability and version control), ensure that the modeling technique can support the service level agreement (SLA) of the AI system, and plan for deployment and maintenance.

Serving Infrastructure: Our previous article mentioned the need to “walk before running” in the development of AI products.  The foundation of any data product consists of “solid data infrastructure, including data collection, data storage, data pipelines, data preparation, and traditional analytics.” A product manager for this phase prepares the way for putting products into production by building the infrastructure needed to support the design, development, and use of future products. This includes tools for model development (such as the Cloudera Data Science Workbench, Domino Data Lab, Data Robot, and Dataiku) and production serving infrastructure (such as Seldon, Sagemaker, and TFX).

Companies have widely different practices, so the roles that AI PMs play varies substantially. Therefore, it’s a good idea to develop some competence in all of these core capabilities.  As the field, technology, and individual organizations mature, specialization will become both necessary and common. In a large company, product management may change hands several times as a product moves through the pipeline. There may be a “product owner” who has end-to-end responsibility for the product’s development. In a small company, a single PM may shepherd a product from conception to operation.

Consumer Companies Versus B2B Companies

It’s not surprising that the company’s business model has a huge effect on the product manager’s work.  Not only are the product’s raw components vastly different in different types of businesses (data, technology infrastructure, and talent), the types of AI products required to serve the customer also differ.

In consumer companies, product managers are more likely to align directly with a feature team, and have much more customer-driven work.  Because they are building an AI product that will be consumed by the masses, it’s possible (perhaps even desirable) to optimize for rapid experimentation and iteration over accuracy—especially at the beginning of the product cycle.  This means that AI PMs must be more hands-on during the experimentation and research phases; it’s their responsibility to align the customer’s voice and needs with research goals.

In addition, product managers at consumer companies often have clearer technical problems to solve.  Many peers or competitors have already created AI products, resulting in ML/AI techniques that are far more mature than in other areas.  For example, product managers for companies that buy or sell advertising are working in a well-researched algorithmic environment and data ecosystem where the emphasis is less on software engineering and more on the development of novel modeling techniques that will move the needle on product outcomes.

The disadvantage of working in a consumer company⁠—especially one that is just getting started⁠—is that there is often a problem with data volume. Modeling techniques that serve interventions to customers rely on detailed demographic information.  The need for specific types of training data is a major challenge. Organizations often find themselves without enough data to determine which experiments to run or which data to obtain. The process of getting the right data can take a long time, and usually goes something like this: you start to build something, ask questions about the data you need, realize you don’t have the right data, start collecting the data (or retrofitting old data), and finally do the analysis and build the product you wanted at the start. To shorten this lengthy cycle, product managers must bring qualitative means of decision making to the table, and should not expect Data Scientists or ML Engineers to have all of the answers.

In contrast, AI product managers working in Business to Business (B2B) firms tend to focus on the first and last mile of the AI product cycle.  B2B firms solve highly complex problems for a very narrow set of consumers.  Take security: many AI/ML-enabled security firms are solely focused on application threat and anomaly detection.  Although the companies they serve may be very diverse, the firms providing these AI products have a clear focus on one or two product types—an advantage that consumer AI products rarely have.

These companies often have access to a lot of data at the beginning of the development cycle—also unlike consumer products.  However, it may not be easy to access or contextualize this data, especially in enterprises.

Once the data challenges are resolved, the model development cycle may prove intractable.  Consider threat detection again: even if we find a significant number of identifiable threats within the dataset, current ML techniques for time series anomaly detection are notoriously difficult to tune. The product manager needs to decide on a technique that meets the precision levels required by businesses, but is interpretable enough to explain and maintain over a product lifecycle.

Finally, integrating AI products into business tech stacks (especially in enterprises) is nontrivial. PMs in B2B firms can’t afford to ignore the stack with which their products will be deployed, nor can they ignore the problems of designing for scale.

Startups Versus Large Companies

Product managers for AI have very different roles and responsibilities in small and large companies.  Large organizations tend to have a lot of data, but that data is usually complex, older, and stored on less flexible (and harder to integrate) technology than in smaller companies. Enterprise data may be logically (or physically) separated into silos, and development of a consistent, cross-enterprise data platform may be a high priority. AI product managers will be more involved in the data products, platform conversations, and project management than they would be in a startup environment.

In enterprises, AI product managers may evolve out of the need to coordinate and manage several cross-functional teams that have developed organically (i.e., data platform, metrics, ML/AI research, and applied ML). One benefit of this evolutionary growth is that enterprise AI PMs will be able to rely on cross-functional domain experience and existing processes from day one, in contrast to those working in startups.  The tradeoff for that collaboration and support is speed of execution and flexibility.  According to VentureBeat, fewer than 15% of Data Science projects actually make it into production.  The number of projects that actually add value (especially in an enterprise context) is probably even lower.  Lack of alignment on a coherent overall data strategy, a focus on technology over impact, an inability to embrace an iterative, experimentational development cycle and lack of leadership support are among the many reasons AI projects falter. Most of these factors are inherent to or exacerbated by the enterprise environment.  AI PMs may have a narrower, less technical focus in large organizations, but the stakes are no lower and the challenges are certainly not simpler or easier to tackle.

In contrast, in startups it’s unlikely that AI PM will be a distinct role, unless one or more products are central to the overall business model (for example, AdTech or search).  As a result, product managers from other functions may find that they need to adopt the roles and responsibilities of an AI PM in their own product areas.  Lack of a specific role definition doesn’t prevent success, but it does introduce the risk that technical debt will accumulate as the business scales.  It is important that an organization’s overall data strategy include waypoints (which may be the stages in the product pipeline) that mark the appropriate time and conditions for upgrading AI resources, technology, and leadership.  This responsibility falls to executive leadership.  Strong AI product management and engineering leadership cannot thrive without support from the C-suite.

In startup environments, the lack of data, the relative immaturity of artificial intelligence and machine learning, the platform environment, and access to AI talent precludes more ambitious projects.  This is both an advantage and a disadvantage! AI PMs in startups enjoy the benefits of flexibility, velocity, and rapid experimentation that enterprise AI PMs could only dream of.  When managed properly, AI products in small firms can add value for the customer and the business almost immediately, and customer feedback can be integrated rapidly.  However, as the business scales (or if the original AI product requires significant cross-team coordination), the responsibilities of the startup AI PM can quickly prove to be overwhelming, requiring negotiation between different teams with different goals, objectives, metrics, and responsibilities.

For example, consider a company that aims to build and sell an AI-enabled personal finance app.  The product’s core features, predicting users’ most common financial planning, banking and expense activities, then executing them automatically, are simple enough when the user population is low and the set of actions that the product must support is small.  A single PM can manage the roadmap for the entire product, including the core model, data platform, APIs, and UI/X.  Now consider the same product as it reaches its 100,000th user and expands to its first international city.  It’s highly unlikely that the same ML model will generalize to the growing user population, and it’s nearly impossible that the same APIs and other integrations will scale globally to support international use. Usage will be different; what people expect from financial institutions will be different; regulation will be different. Organizations that can successfully navigate the transition from startup to enterprise AI are the ones that carefully consider the skill set and experience of the AI PM at each stage of growth.

The Data Expertise of the AI PM

Product Managers are expected to bring a cross-functional skill set to the table, so that they can support all aspects of bringing a product to market and supporting it through its lifecycle.  Some product managers may be more technical, perhaps with a background or education in software engineering; others may specialize in design, customer success, UI/UX, or some other aspect of product development. Product managers for AI must be able to support their products throughout the entire pipeline, and as a result some expertise in each of the key categories is required.  It is not our aim to provide an exact specification for the skills that will ultimately make an AI PM successful, but rather to identify the minimum viable skill set necessary to support the AI product lifecycle.

Skill-Data Lifecycle and Pipeline Management: No AI product can succeed without quality data. AI PMs must learn to operate in an environment where the economics and resource constraints inherent to obtaining data, processing it for use in experiments and customer-facing AI products, and ensuring quality over time are seldom favorable.  At a minimum, AI PMs must understand the vocabulary of this space and be able to contribute to platform decisions that will impact AI products downstream.

Skill-Experimentation and Measurement: Whether through exploratory experimentation, pre-deployment A/B testing, or post-deployment evaluation of adoption and engagement, AI PMs must be excellent designers of experiments and experts at interpreting experiment results. The minimum viable skill set in this area includes a basic understanding of probability theory (distributions, cohorting, confidence, power, etc.), a deep understanding of A/B testing, and a similarly deep knowledge of model evaluation techniques. Avinash Kaushik’s Web Analytics 2.0 is an excellent introduction to metrics and analytics.

Skill-DS/ML/AI Development Process: At a minimum, Software Engineering PMs should be fluent in the processes and language of effective software development.  They should be familiar with agile software development practices, continuous integration and continuous deployment (CI/CD), and the principles of DevOps.  AI PMs should possess some degree of expertise in development processes for data science and machine learning, such as CRISP-DM, the Microsoft Team Data Science Process (TDSP), or Continuous Delivery for Machine Learning. If the AI product manager doesn’t have a background in software products, they should be sure to enlist the support of a product manager who does. The point isn’t so much what process you use, but to have a process.

Conclusion

Nobody has all of the skills at the same time, so get to work building the ones you need. There are many resources available for people who want to develop AI skills: blogs, papers, competitions, and courses, all both paid and free. A product manager for AI doesn’t have to be an expert in everything–or even in anything.  But a successful product manager does need to have a broad view of how AI products are built, from start to finish.

In our next article, we’ll look in more detail at the development process and the product manager’s responsibilities at each stage of that process.