ML experiment tracking in 2026: revisit MLflow, Aim and ClearML before you buy another hosted tracker seat

Many ML and LLM teams still treat a hosted experiment tracker as the adult default. A few notebook charts feel messy, someone pastes a Weights & Biases or Neptune signup link into Slack, and the team buys another seat before anyone asks where the run history, artefacts and model lineage must live. That path can be right. It is not the only path, and it is a poor default when you have not asked what you need to operate.

Self-hosted experiment tracking keeps parameters, metrics, artefacts and (when you use them) model registry records on infrastructure you control. Training and eval code still runs where it already runs. The difference is that the shared UI and metadata store sit on your SQL database, object store or local repo, not only in a vendor cloud. This post revisits three options teams still meet when they want that model without renting another hosted tracker by habit: MLflow, Aim (AimStack branding; repository aimhubio/aim), and ClearML. Weights & Biases and Neptune appear only as the hosted habit to unlearn, not as peers. The aim is not to crown a winner. The aim is to help you pick something you can operate.

The short version

• MLflow: Linux Foundation-backed Apache-2.0 platform (Databricks origin) for tracking, model registry and growing GenAI/agent tooling. Strong default when you will own a Tracking Server, SQL backend, artefact store and auth.
• Aim: Lightweight Apache-2.0 experiment tracker with a local .aim repo (RocksDB plus SQLite metadata) and a strong compare UI. Strong for individuals and small teams on a trusted network; release cadence has been quiet since May 2025.
• ClearML: Full MLOps suite (experiment manager, agents, datasets, serving). SDK, agent and web UI are Apache-2.0; the server is SSPL. Strong when you need queues and remote workers, and you accept a multi-container ops bill plus the SSPL boundary.

What experiment tracking and an MLOps platform actually do

Before comparing tools, it helps to be clear about the job. A practical experiment tracker or MLOps platform usually covers some mix of these:

• Log a run: parameters, metrics, tags, code or environment fingerprints, and artefacts (checkpoints, plots, prompts, eval dumps).
• Compare runs in a UI: filter experiments, overlay curves, and answer “which config actually won” without a shared spreadsheet.
• Optionally register models: promote a logged artefact to a named model version with stage or alias rules.
• Optionally orchestrate remote jobs: enqueue training on GPU workers, Kubernetes or cloud VMs from a shared control plane.
• Keep experiment data closer to home: metadata and artefacts on storage you can back up, audit and delete under your retention policy.

It is not magic. You still own upgrades, auth, network exposure, backups and the discipline of logging consistently from every training entrypoint. The win is that you stop treating a hosted tracker seat as the only adult option, and you stop losing lineage every time a laptop is wiped.

MLflow: tracking, registry and GenAI on Apache-2.0

History and licence. The project lives at mlflow/mlflow. The repository was created in June 2018. The LICENSE file carries Apache License 2.0 with a Databricks copyright header. In June 2020 Databricks announced that MLflow was joining the Linux Foundation; mlflow.org still describes the project as backed by the Linux Foundation. Treat that governance story and the managed Databricks MLflow / Unity Catalog path as related but separate: the open-source default is what you self-host.

How it works. Classic MLflow centres on runs grouped into experiments, plus a model registry for promoted artefacts. Without a server, metadata and artefacts land under a local ./mlruns directory. For a shared team setup, you run a Tracking Server, point the backend store at a SQLAlchemy database (SQLite for trials; PostgreSQL or MySQL for teams), and put artefacts on a local path, S3, MinIO, Azure Blob or GCS. Docs describe auth options including basic HTTP auth, SSO and plugins, and warn that the built-in server should not be broadly exposed unauthenticated or unencrypted. APIs cover Python, REST, R and Java, with autolog hooks for many libraries. Current upstream messaging also positions GenAI observability (tracing), evaluation, prompt tooling, an AI Gateway and agent-oriented features alongside classic MLOps. Keep this post’s comparison on experiment tracking and self-host ops; treat LLM features as expansion, not as the only reason to adopt MLflow.

At the time of writing, GitHub and PyPI show v3.18.0 (4 November 2026) as the latest stable line we checked, with active 3.17.x and 3.16.x releases through autumn 2026. Release notes for 3.18.0 highlight automatic issue detection for agent failures, an experimental Skill Registry, revamped evaluation and trace comparisons, and experimental online evaluation that needs self-hosted job containers. Pin a version, read the release notes, and plan upgrades like any other production service.

Where it is strong.

• Teams that want one Apache-2.0 stack for tracking plus model registry, with a clear path to SQL and object-store backends.
• Groups that already speak MLflow client APIs, or that want autolog and a widely documented remote-server tutorial.
• Organisations that expect GenAI tracing and evaluation to grow next to classic training runs, without leaving the same platform family.
• Platform teams that can own auth, reverse proxy, private networking and upgrade cadence.

What to watch.

• Default or open Tracking Servers on a public interface are unsafe. Upstream docs warn about unauthenticated and unencrypted exposure; put the UI and API behind auth, TLS and a private network.
• Self-host security has been an active topic in 2026. Public advisories for the MLflow project include identifiers such as CVE-2026-116692, CVE-2026-116690, CVE-2026-115886 and CVE-2026-64849, plus older critical classes such as CVE-2023-1177. Cite the IDs, upgrade promptly, and do not treat “it runs on port 5000 in the lab” as a production design.
• Managed Databricks MLflow is a commercial escape hatch, not the same ops story as self-hosting PostgreSQL and S3 yourself.
• Scope creep: if you only need scalar charts for one researcher, a full team Tracking Server may be heavier than Aim for that narrow job.

Aim: lightweight compare UI on a local .aim repo

History and licence. The project lives at aimhubio/aim. The repository was created in May 2019. It is licensed under Apache License 2.0. Product branding uses AimStack (site aimstack.io); the canonical GitHub repository remains aimhubio/aim. Do not confuse branding with a repository rename. The package is fully self-hosted; there is no separate SSPL server for the core tracker.

How it works. pip install aim gives you the SDK, UI and CLI in one package. aim init creates a local **.aim** repository. You log with aim.Run / Run.track and open the UI with aim up (default http://127.0.0.1:43800). Storage docs describe RocksDB containers under .aim plus run_metadata.sqlite for structured metadata, designed so basic local logging does not need extra services. For multi-host use, upstream documents aim server (default port 53800) and client repos of the form aim://host:53800, with optional SSL key and cert flags. Docker images expose the same UI and server ports. Integrations listed in the docs cover common training stacks (PyTorch Lightning, Hugging Face, TensorFlow/Keras, XGBoost and others).

Aim is an experiment and metadata tracker with a strong compare UI. It is not a full job orchestrator, agent queue or model-serving suite. If you need remote GPU enqueue and pipelines in one product family, look at ClearML instead of stretching Aim.

Release reality (be honest). The latest GitHub release page and PyPI version we checked are v3.29.1 (8 May 2025). Earlier 3.28.x and 3.27.x releases sit in early 2025 and late 2024. Tags for a 4.x line exist historically, but they do not define the current PyPI product; do not call Aim 4 the shipping tracker. Main-branch activity after the May 2025 release has been quiet relative to MLflow and the ClearML SDK in 2026. Treat maintenance as slow and quiet, not as a declaration that the project is abandoned. Plan for that: pin 3.29.1, test upgrades carefully, and keep the UI off the public internet.

Where it is strong.

• Individuals and small teams that want a polished run-compare UI with data on disk or on a VPN, without standing PostgreSQL on day one.
• Groups that prefer a single Python package and an embedded storage layout over a multi-service compose stack.
• Engineers who care more about querying and comparing metadata sequences than about a central model registry.

What to watch.

• Quiet release cadence means security fixes may lag. Several GitHub Advisory entries for the aim PyPI package list first_patched_version as None even when the vulnerable range includes current 3.29.1. Examples include CVE-2025-5321, CVE-2025-51464 and older critical path issues such as CVE-2024-8769. Do not claim “all CVEs are fixed in 3.29.1” when advisory metadata contradicts that for at least some IDs.
• Keep Aim on trusted networks. Check advisories before any internet exposure; prefer VPN, SSO at a reverse proxy, and least privilege.
• No built-in model registry or agent orchestration. Pairing Aim with a separate registry or job system is fine; pretending Aim is ClearML is not.
• Enterprise support contact on the README is vendor support, not a second licence for the core.

ClearML: MLOps suite with an Apache SDK and an SSPL server

History and licence. The client SDK lives at clearml/clearml (organisation formerly associated with Allegro Trains / allegroai; treat ClearML as the current name). Related repos include clearml-server, clearml-agent and clearml-web. Licence split matters. The SDK, agent and web UI are Apache-2.0. The server is Server Side Public License v1.0 (SSPL), with copyright on the LICENSE file naming ClearML Inc. ClearML’s open-source terms page (effective 1 February 2026) restates that split: Apache-2.0 for SDK/UI, SSPL for the server. SSPL section 13 is broad if you offer the program’s functionality to third parties as a service. Internal self-host for your own organisation is the usual documented path. Do not flatten the story to “ClearML is Apache open source.”

How it works. Upstream describes a suite: Experiment Manager, MLOps/LLMOps via clearml-agent, data management for datasets on object storage, model serving (clearml-serving), reports and orchestration dashboards. The self-hosted server, via official Docker Compose, is a multi-service stack: web UI (port 8080), API (8008), file server (8081), plus MongoDB, Elasticsearch and Redis. Compose pins observed for this research aligned with server v2.4.0 notes (Elasticsearch 8.19.9 among other fixes). Deploy paths also include Kubernetes Helm charts and cloud images. After compose is up, you create credentials, point the SDK at your server (or at the hosted app.clear.ml service), and optionally install clearml-agent on GPU or CPU workers so the UI can enqueue tasks onto queues.

At the time of writing, the SDK release we checked is v2.2.0 (3 November 2026) on GitHub and PyPI, with steady 2.x patches through 2026. The server tag we checked is v2.4.0 (8 March 2026). The agent package sits on its own release line (PyPI clearml-agent 3.0.3 among recent tags). Pin each component deliberately; do not assume one version number covers SDK, server and agent.

Where it is strong.

• Teams that want tracking plus remote execution, queues and pipelines on their own GPUs or Kubernetes nodes in one product family.
• Groups that already accept operating MongoDB, Elasticsearch and Redis for other platforms, or that prefer ClearML’s hosted control plane while keeping workers on-prem.
• Organisations that need dataset versioning and serving helpers next to experiment history, not only scalar charts.

What to watch.

• Heaviest self-host footprint of the three. Budget for compose or Helm, disk for Elastic/Mongo, TLS, backups and upgrade rehearsal.
• SSPL on the server: run legal review before any multi-tenant or “ClearML as a service” idea; internal use is the common OSS path.
• OSS versus paid feature gaps belong on ClearML’s own pricing and product pages. Treat that matrix as the vendor’s own comparison, not as an independent benchmark in this post. We do not invent seat prices or ROI percentages here.
• Security: SDK advisory CVE-2025-8917 (path traversal in safe_extract) is listed as patched in 2.0.2; current 2.2.0 is past that patch. Older Allegro-era issues exist in public CVE lists; prefer “run current 2.x SDK, harden the server, private network” over inventing a clean 1.x fix line.
• Do not expose 8080/8008/8081 on the public internet without auth and TLS.

Side by side, without a table

Primary job

• MLflow: experiment tracking + model registry, with expanding GenAI/agent tooling on the same platform.
• Aim: experiment/metadata tracking and compare UI; not a registry or orchestrator.
• ClearML: experiment manager plus agents, datasets, pipelines and serving in one suite.

Licence (as stated upstream)

• MLflow: Apache-2.0 for the project.
• Aim: Apache-2.0.
• ClearML: Apache-2.0 for SDK, agent and web; SSPL for the server.

Self-host weight

• MLflow: medium for teams (Tracking Server + SQL + object store + auth).
• Aim: lightest (embedded RocksDB / .aim; optional remote aim server).
• ClearML: heaviest (MongoDB + Elasticsearch + Redis + API + UI + file server).

Release cadence checked for this post (November 2026)

• MLflow: v3.18.0 (4 November 2026); very active 3.x line.
• Aim: v3.29.1 (8 May 2025); quiet since then relative to peers.
• ClearML: SDK v2.2.0 (3 November 2026); server v2.4.0 (8 March 2026); agent on its own 3.x line.

Beyond scalar charts

• MLflow: model registry, evaluation helpers, GenAI tracing/gateway/agent features.
• Aim: strong compare UI and programmatic query over run metadata.
• ClearML: clearml-agent queues, datasets, serving, orchestration dashboard.

Hosted escape hatch

• MLflow: Databricks managed MLflow and related commercial paths.
• Aim: self-host oriented.
• ClearML: app.clear.ml hosted service plus enterprise platform messaging.

We did not run install drills, load tests or security PoCs for this post. Prefer the official install guides on disposable infrastructure before you move a whole ML org.

A practical way to start this week

For an Indian engineering team that is about to renew a hosted tracker seat or drown in “which notebook had the best F1” chat threads, a calm sequence looks like this:

1. Write down the real constraints: solo laptop vs shared team UI; need a remote GPU queue; must metadata and artefacts stay on-prem; is an SSPL server acceptable to legal.
2. Spike MLflow locally (pip install mlflow or an equivalent pinned install), start mlflow server on a loopback port, log one autolog or manual run, open the UI.
3. Spike Aim on the same toy script (pip install aim, aim init, log with Run, aim up). Compare how it feels to filter and overlay runs.
4. Only if orchestration matters, stand ClearML Server with the official Docker Compose path on a private VM, create credentials, log a Task, and optionally attach one clearml-agent worker. Do not publish 8080/8008/8081 to the open internet.
5. For whichever option you keep for shared use: put the UI behind SSO or VPN, back up the metadata store and artefact bucket, and schedule upgrades after reading current advisories.
6. Revisit in 90 days: are people using registry, agents and datasets, or only scalar charts? Shrink or expand the footprint accordingly.

How to choose

• If you want the common Apache-2.0 standard with registry and an active release train, and you will own SQL, object storage and auth, start with MLflow.
• If you want the smallest self-host footprint and a strong compare UI, and you can accept a quiet release train plus strict network isolation, start with Aim.
• If you need tracking plus agents, queues and pipelines on your GPUs, and legal accepts SSPL for the server, start with ClearML.
• If the only pain is “we need somewhere to dump curves this sprint”, do not buy a full MLOps suite on day one. Prove logging discipline on MLflow or Aim first.
• Do not run MLflow, Aim and ClearML “for a month” on the same production GPU cluster without clear owners. Compare on throwaway projects.

Mistakes to avoid

• Signing up for another hosted tracker seat because Slack already has the invite link, before you write down residency, backup and cost ownership.
• Exposing MLflow, Aim or ClearML UI/API ports on a public IP with default or empty auth.
• Calling ClearML “Apache open source” without mentioning the SSPL server.
• Treating Aim 4.x tags as the current product, or calling Aim abandoned because releases slowed after May 2025.
• Assuming advisory IDs are fixed because you installed “latest” once; check patched versions, especially where Aim advisories list no patch.
• Buying ClearML agents and serving modules when the team only logs two scalars a week.
• Skipping backups of the metadata database and artefact store until the first disk failure.
• Mixing Databricks-managed MLflow docs with self-hosted Tracking Server runbooks without noticing the difference.

Checklist before you commit

• Are we renewing a hosted tracker because we measured shared lineage needs, or because notebook folders felt messy last quarter?
• Must run history and artefacts stay in our VPC or on-prem object store?
• Have legal reviewed Apache-2.0 (MLflow, Aim, ClearML SDK) versus SSPL (ClearML server) for our use case?
• Who owns upgrades and advisory response for the chosen stack?
• Is the UI reachable only through VPN, SSO or a private network path?
• Do we have a restore drill for metadata and artefacts, not only a screenshot of the dashboard?
• Will we use model registry or remote agents in the next two quarters, or only compare curves?
• Can a new teammate log a run without receiving a shared tracking URI and password in chat?
• Have we pinned versions (MLflow 3.18.x, Aim 3.29.1, ClearML SDK/server/agent separately) instead of floating latest?

Closing

Treating every training run as disposable notebook output is a habit worth unlearning. Buying another hosted experiment tracker seat without asking where lineage must live is the same habit in a nicer UI. Revisit MLflow, Aim or ClearML with the mindset that fits this site: learn what the tracker actually stores, unlearn the assumption that experiment history must be a rented product, re-learn the licence and security boundaries, practise on disposable infra, and apply the chosen tool only where it reduces real toil.

Sources

• MLflow repository: https://github.com/mlflow/mlflow
• MLflow v3.18.0 release: https://github.com/mlflow/mlflow/releases/tag/v3.18.0
• MLflow on PyPI: https://pypi.org/project/mlflow/
• MLflow site and docs: https://mlflow.org/ and https://mlflow.org/docs/latest/
• MLflow tracking docs: https://mlflow.org/docs/latest/ml/tracking/
• MLflow remote server tutorial: https://mlflow.org/docs/latest/ml/tracking/tutorials/remote-server/
• MLflow self-hosting docs: https://mlflow.org/docs/latest/self-hosting/
• Databricks blog: MLflow joins the Linux Foundation (25 Jun 2020): https://www.databricks.com/blog/2020/06/25/mlflow-joins-the-linux-foundation-to-become-the-open-standard-for-machine-learning-platforms.html
• MLflow security advisories: https://github.com/mlflow/mlflow/security/advisories
• Aim repository: https://github.com/aimhubio/aim
• Aim v3.29.1 release: https://github.com/aimhubio/aim/releases/tag/v3.29.1
• Aim on PyPI: https://pypi.org/project/aim/
• AimStack site and docs: https://aimstack.io/ and https://aimstack.readthedocs.io/en/latest/
• Aim setup, storage and remote tracking docs: https://aimstack.readthedocs.io/en/latest/quick_start/setup.html , https://aimstack.readthedocs.io/en/latest/understanding/data_storage.html , https://aimstack.readthedocs.io/en/latest/using/remote_tracking.html
• ClearML SDK repository: https://github.com/clearml/clearml
• ClearML SDK v2.2.0 release: https://github.com/clearml/clearml/releases/tag/v2.2.0
• ClearML server repository: https://github.com/clearml/clearml-server
• ClearML server v2.4.0 release: https://github.com/clearml/clearml-server/releases/tag/v2.4.0
• ClearML agent and web repositories: https://github.com/clearml/clearml-agent and https://github.com/clearml/clearml-web
• ClearML on PyPI: https://pypi.org/project/clearml/ and https://pypi.org/project/clearml-agent/
• ClearML site and server deploy docs: https://clear.ml/ , https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server , https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_linux_mac
• ClearML open-source terms: https://clear.ml/legal/open-source/terms
• ClearML hosted service: https://app.clear.ml
• SSPL reference text (MongoDB): https://www.mongodb.com/legal/licensing/server-side-public-license