CLI Companion

  • Hugging Face CLI
    • login
    • whoami
    • repo create
    • upload
    • download
    • lfs-enable-largefiles
    • scan-cache
    • delete-cache
  • Hapi CLI
    • new
    • start
    • build
    • test
    • plugin create
    • route add
  • Cloudflared
    • tunnel
    • tunnel run
    • tunnel list
    • tunnel delete
    • access
    • access tcp
    • update

    The `huggingface-cli lfs-enable-largefiles` command is a utility within the Hugging Face CLI designed to simplify the process of enabling and configuring Git Large File Storage (LFS) for your local repositories, particularly when interacting with the Hugging Face Hub. It ensures that large files, such as model weights, datasets, and other binaries, are handled efficiently by Git LFS rather than standard Git, which is not optimized for large binaries.

    Syntax

    bash
    huggingface-cli lfs-enable-largefiles [REPOSITORY_PATH]

    * `REPOSITORY_PATH` (optional): The path to the local Git repository where you want to enable LFS. If not provided, the command will default to the current working directory.

    Explanation

    When you work with machine learning models and datasets, you often encounter files that are hundreds of megabytes or even several gigabytes in size. Standard Git struggles with these large files, leading to slow operations, bloated repository histories, and difficulties with cloning and pushing.

    Git LFS addresses these issues by storing pointers to the large files in your Git repository while the actual file contents are stored on a separate LFS server (like the Hugging Face Hub's LFS storage). The `huggingface-cli lfs-enable-largefiles` command automates the setup for this process:

    1. **Initializes Git LFS**: If Git LFS has not already been initialized for the repository, it will run `git lfs install` to set up the necessary Git hooks.

    2. **Configures `.gitattributes`**: It automatically adds or updates a `.gitattributes` file in your repository. This file tells Git which file types should be tracked by LFS. The Hugging Face CLI command is smart and typically pre-configures common large file extensions relevant to machine learning, such as:

    * `*.bin` (for PyTorch, TensorFlow, etc.)

    * `*.pt` (PyTorch model files)

    * `*.safetensors` (SafeTensors files)

    * `*.h5`, `*.hdf5` (HDF5 files, common in Keras/TensorFlow)

    * `*.onnx` (ONNX model files)

    * `*.msgpack`, `*.arrow`, `*.parquet`, `*.tfrecord` (common dataset formats)

    * And other similar large binary formats.

    Usage Examples

    #### 1. Enable LFS in the current directory

    If you are already inside your local Git repository, you can simply run the command without specifying a path:

    bash
    cd my-model-repo
huggingface-cli lfs-enable-largefiles

    This will configure LFS for `my-model-repo`, making it ready to track large files.

    #### 2. Enable LFS for a specific repository by path

    If you want to enable LFS for a repository that is not your current working directory, you can provide its path:

    bash
    huggingface-cli lfs-enable-largefiles /path/to/my/another-model-repo

    This is useful for scripting or when managing multiple repositories from a central location.

    #### 3. Verifying LFS configuration

    After running the command, you can inspect your `.gitattributes` file to see the newly added LFS tracking rules. For example, it might contain lines like:

    *.bin filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text

    You can also check if LFS is working correctly by trying to add a large file and inspecting its status:

    bash
    echo "large file content" > large_model.bin # Create a dummy large file
git add large_model.bin
git status

    If LFS is enabled, `git status` won't show the full content of `large_model.bin` as a change, and when you commit and push, Git LFS will handle the actual file transfer.

    Important Considerations

    * **Install Git LFS**: Ensure Git LFS itself is installed on your system *before* running this command. You can usually install it via your system's package manager (e.g., `sudo apt install git-lfs` on Debian/Ubuntu, `brew install git-lfs` on macOS, or download from [git-lfs.github.com](https://git-lfs.github.com/)).

    * **Run Early**: It is crucial to run `huggingface-cli lfs-enable-largefiles` *before* you add and commit any large files to your repository. If you commit large files with standard Git first, and then enable LFS, those files will remain in your Git history as large objects, potentially requiring complex history rewriting to fix.

    * **Manual `.gitattributes`**: While the command provides sensible defaults, you might need to manually edit the `.gitattributes` file to add or remove specific file types that should or should not be tracked by LFS based on your project's needs.

    * **Hugging Face Hub Integration**: When you push a repository with LFS-tracked files to the Hugging Face Hub, the Hub automatically stores these files using its LFS infrastructure, making model and dataset sharing seamless.