Forbidden Secrets of the Metadata API

Salesforce’s Metadata API is a powerful resource that allows developers and system administrators to manage complex change and release processes across large organizations. I first began working with this API more than a decade ago when my Snapshot product was initially published. Over the years, I have grown to appreciate the Metadata API’s capabilities, but I’ve also encountered a variety of special quirks that can trip up unsuspecting users.

In this paper, we will examine some of the most frequently encountered technical questions about the Metadata API. We will focus on three core operations:

• Retrieve API: Pulls assets from an org and places them into a local directory (often used to back up or compare metadata).
• Deploy API: Pushes assets into an org, thereby updating or adding metadata components.
• List Metadata API: Lists which assets are present in an org, providing a blueprint of the metadata hierarchy.

At first glance, these tasks appear straightforward. Yet, each has hidden complexities that demand caution and understanding. So, buckle up and get ready to explore these “forbidden secrets” of the Metadata API.

Deploying Managed and Unmanaged Packages

Many people are unaware that both managed and unmanaged packages can be deployed using the Metadata API. However, the process can be more complex and nuanced than handling “unpackaged” metadata components:

• No Wildcards for Unmanaged Packages: When deploying an unmanaged package, you must explicitly list every asset and child asset type in your package.xml. Wildcards (such as *) will not work in the Deploy API for unmanaged packages, so you need to be meticulous in enumerating desired components.
• Specify Full Name: In addition to listing every asset, you must also include the “fullName” parameter for each. If done correctly, unmanaged packages can be moved between orgs relatively easily.
• Beware of Mixing Child Types with Wildcards: Even when handling unpackaged assets, mixing wildcards for parent objects while specifying child objects explicitly can trigger errors. It’s generally safer to avoid using wildcards when you are trying to deploy multiple related asset types together.

For managed packages, the deployment flow is more unusual:

• Installed Packages: Each installed managed package in your org is identified by its namespace within the InstalledPackages metadata type. Deploying a single Installed Package metadata entry will begin the installation process and even send out an email notification, just like you clicked “Get It Now” on AppExchange.
• Deploy Alone: Deployed managed packages should be installed by themselves. Attempting to bundle additional metadata with that package’s deployment often results in unexpected conflicts or failures.

Strange List Metadata Results

One of the most puzzling methods of the Metadata API is the List Metadata call. It returns a collection of asset names for any metadata type you request. This seems straightforward, but the results are often incomplete or inconsistent. For instance:

• Custom Objects: You get names of all custom objects, packaged objects, and some standard objects, requiring special naming conventions to differentiate them.
• Standard vs. Custom Tabs: “standard-” prefixed tabs are sometimes omitted from the response, yet “standard_” prefixed applications might still appear, leading to confusion.
• Profiles: The names you retrieve may not match the textual screen names found in the Salesforce UI. You often have to decipher naming differences.
• Permissions: List Metadata includes Custom Permissions but does not include specific User Permissions.

Additionally, List Metadata provides inaccurate naming conventions for Page Layouts—a challenge we address next.

Page Layout Names and Managed Packages

Page Layouts in the Metadata API use a naming pattern of [object name]-[layout name]. When dealing with packages, the pattern becomes more complicated:

• Std Object & Unpkg Layout: Account-mylayout
• Cust Object & Unpkg Layout: myobj__c-mylayout
• Pkg Object & Unpkg Layout: myspace__myobj__c-mylayout
• Std Object & Pkg Layout: Account-myspace__mylayout
• Cust Object & Pkg Layout: myobj__c-myspace__mylayout
• Pkg Object & Pkg Layout: myspace__myobj__c-myspace__mylayout

The tricky part is that List Metadata often returns what looks like an “unpackaged” layout name even if the layout actually belongs to a managed package. To successfully retrieve and deploy the correct layout, you must construct the real name, including the relevant namespace after the dash. Importantly, you cannot simply rely on the presence or absence of a namespace prefix before the dash. Layouts are unique in how they embed the package namespace.

Standard Assets Complicate Deployment

Deploying metadata quickly gets complicated due to interdependencies among custom assets—Profiles, Permission Sets, Page Layouts, and more. Recently, Salesforce has introduced new types of orgs with their own “standard” metadata elements such as unique fields, tabs, and apps:

• New Standard Fields: Some orgs have automatically included standard fields like CleanStatus when Data.com is activated. These can break deployment if referenced in your source org and missing in the destination org.
• References in Profiles and Layouts: Standard fields or user preferences can appear everywhere, from Profiles to Permission Sets. You cannot create these standard assets if they do not already exist in the target org, leading to errors.
• Difficult to Detect: You often discover these references only when your deployment fails. The List Metadata API does not provide a simple route to find every possible standard field or user preference that might cause conflicts.

Tools like Snapshot can help remove or clean up these missing references automatically, but some manual intervention is often still required.

Canonical Sorting and Metadata Differences

Many of the XML documents retrieved via the Metadata API come in canonical order, meaning their elements are alphabetically sorted. Meanwhile, other XML documents rely on specific element order or appear scrambled:

• Canonical Examples: Profiles typically list child elements (permissions, field-level security, etc.) in alphabetical order.
• Order-Sensitive Examples: Picklists in Standard or Global Value Sets rely on a particular arrangement of elements. Changing that order can alter the meaning or cause potential conflicts.
• Random Extraction: Certain metadata types like Clean Data Services or Site Dot Coms can appear in a different order each time you retrieve them, making version comparisons tedious.

When performing metadata comparisons across multiple orgs or different points in time, canonical sorting can help reduce false positives in version control. However, you still need the unsorted files to properly deploy your changes.

Sparse Matrix Considerations in Profiles

Profiles in Salesforce can have large child segments, but many of these segments are sparse. For example, Tab Visibility, Object Permissions, User Permissions, and Custom Permissions only list items that are turned on or granted. How does this affect deployments?

• Merge Behavior: When you deploy a Profile, certain elements (especially child types that are sparse) can merge into the existing metadata at the destination. If something is disabled on the source, it won’t appear in the XML, so you might not be able to turn off a permission without explicitly adding an override element.
• Permission Sets vs. Profiles: Permission Sets also use a sparse model but replace the destination metadata outright. Consequently, turning off a permission in a Permission Set can be done by simply excluding it.
• Layout Assignments: Profile Layout Assignments behave like Permission Sets: they are sparse but they replace whatever is in the destination, which can simplify deployment.

Specialized tools can help you “force” certain permissions to be disabled, or highlight changes that need to be explicitly stated in the deployment to ensure the correct merges happen.

Vanishing Translation Comments

When you retrieve Translations metadata—such as Standard Value Set Translations, Global Value Set Translations, or Object Translations—Salesforce places comments in the XML indicating which text remains unaltered or unlocalized:

Replacing the comments with actual translations and then deploying works perfectly. However, if you retrieve these files and deploy them again unchanged, Salesforce interprets the comment as an empty string and removes it, causing the translation to vanish. The remedy: remove the parent element of any translation still containing commented text before redeploying.

Special Formats for Retrieved Metadata

Not all metadata manifests the same way when retrieved. Most components appear under a folder named for their type, using an XML file to store definitions, but some break this pattern:

• Reports, Dashboards, Email Templates, Documents: Each of these includes subfolders that store files and relevant XML definitions separately.
• Territory2 Model Structure: The Territory2 and Territory2Rule types thread together in a nested series of folders that can be confusing at first glance.
• Settings Type: Uses multiple base element names in its retrieved XML, which can be surprising if you expect all “settings” to appear in one place.
• Assets with Unstructured Data: Wave Templates and Aura Definition Bundles can include a mix of XML and non-XML files. Understanding their directory layout is crucial for smooth migrations.

A thorough grasp of these structures will help you navigate the “metadata jungle” without losing track of scattered assets.

With Great Power

The Metadata API provides Salesforce teams the agility to automate large-scale changes and preserve customizations through version control. Yet, as we’ve seen, it comes with its own eccentricities, from complicated naming conventions to missing references and random XML ordering. I’ve invested years unraveling these mysteries, and I hope this paper helps you avoid the same frustrations.

If managing your org’s day-to-day operations is your highest priority, or if you’re eager to bolster your company’s compliance and security right away, then consider using a specialized tool like Snapshot on the AppExchange. It can save you significant time and effort by automating many of the tricky details we’ve covered. The Metadata API’s power is immense—and with just a bit of additional knowledge, it can transform an otherwise daunting process into a reliable workflow for any Salesforce professional.