GitHub Content Downloader (ghcp)

A powerful, lightweight command-line tool for downloading files and directories from GitHub repositories with ease. Built with Dart, ghcp provides a simple yet flexible way to fetch content from GitHub repositories without cloning the entire repository. Perfect for CI/CD pipelines, automation scripts, and selective downloads.

Features

Fast & Lightweight: Only downloads what you need, no repository cloning required
Directory Support: Download entire directories with preserved structure and recursive processing
Flexible URLs: Supports various GitHub URL formats (blob/tree, different branches)
Recursive Downloads: Automatically handles nested directories with concurrent processing
Concurrent Downloads: Parallel file downloads for improved performance
File Size Awareness: Handles files up to GitHub's 100MB limit

Installation
Quick Start
Usage
Configuration
Supported URL Formats
Output Behavior
Examples
API Rate Limits
Development
Troubleshooting
FAQ
Contributing
License
Acknowledgments

Installation

Option 1: Download Pre-built Binary (Recommended)

Download the latest pre-built binary for your platform from the releases page:

Available Platforms:

macOS: x64 (Intel), arm64 (Apple Silicon)
Linux: x64, arm64
Windows: x64

Linux/macOS:

# Download the appropriate binary for your platform
# Replace {version} with the latest version (e.g., v1.0.0)
# Replace {platform} with: linux or macos
# Replace {arch} with: x64 or arm64

curl -L -o ghcp.tar.gz https://github.com/aminnez/ghcp/releases/download/{version}/ghcp-{version}-{platform}-{arch}.tar.gz

# Extract and install
tar -xzf ghcp.tar.gz
chmod +x ghcp
sudo mv ghcp /usr/local/bin/

# Verify installation
ghcp --help

Windows:

# Download the binary
# Replace {version} with the latest version (e.g., v1.0.0)
Invoke-WebRequest -Uri "https://github.com/aminnez/ghcp/releases/download/{version}/ghcp-{version}-windows-x64.zip" -OutFile "ghcp.zip"

# Extract the zip file
Expand-Archive -Path "ghcp.zip" -DestinationPath "."

# Add to PATH or run directly
# You can move ghcp.exe to a directory in your PATH

Option 2: Install from pub.flutter-io.cn

Prerequisites: Dart SDK Version 3.8.1 or higher (Install Dart)

# Install globally
dart pub global activate ghcp

# Update to latest version
dart pub global activate ghcp

Option 3: Install from Source

Prerequisites: Dart SDK Version 3.8.1 or higher (Install Dart)

# Clone the repository
git clone https://github.com/aminnez/ghcp.git
cd ghcp

# Install dependencies
dart pub get

# Activate globally
dart pub global activate --source path .

Verify Installation

ghcp --help

Quick Start

# Download a single file
ghcp https://github.com/open-webui/open-webui/blob/main/docker-compose.yaml

# Download a single file with custom filename
ghcp https://github.com/open-webui/open-webui/blob/main/docker-compose.yaml compose.yml

# Download an entire directory
ghcp https://github.com/google/dart-basics/tree/master/lib

# Download directory contents directly into target folder
ghcp https://github.com/google/dart-basics/tree/master/lib ./dart-basics

Usage

Command Syntax

ghcp <github-url> [output-directory] [--token <token>] [--help]

Options

<github-url>: GitHub URL to download from (required)
- Supports both blob (single files) and tree (directories) URLs
- Must follow the format: https://github.com/owner/repo/blob|tree/branch/path
[output-directory]: Output directory or filename (optional, defaults to current directory)
- For single files: Can specify a custom filename (e.g., config.yml) or directory
- For directories: Specifies where to extract the directory contents
--token <token>: GitHub personal access token for authentication
-h, --help: Show help message

Authentication

For private repositories or higher rate limits, you can provide a GitHub Personal Access Token:

Using Environment Variable (Recommended)

export GITHUB_TOKEN=your_github_token
ghcp https://github.com/your-username/your-private-repo/blob/main/path/to/file

Using Command Line Flag

ghcp https://github.com/your-username/your-private-repo/blob/main/path/to/file --token your_github_token

Note: Environment variable takes precedence over command line flag.

Configuration

Environment Variables

Variable	Description	Default
`GITHUB_TOKEN`	GitHub Personal Access Token for authentication	`null`

Creating a GitHub Token

Go to GitHub Settings > Developer settings > Personal access tokens
Click "Generate new token"
Select appropriate scopes (at minimum repo for private repositories)
Copy the token and set it as an environment variable

Supported URL Formats

ghcp supports the following GitHub URL formats:

Single Files

https://github.com/owner/repo/blob/branch/path/to/file.ext

Directories

https://github.com/owner/repo/tree/branch/path/to/directory

Root Directory

https://github.com/owner/repo/tree/branch

Output Behavior

ghcp intelligently handles output paths based on the download type and specified output:

Single File Downloads

Custom filename: ghcp <url> config.yml → saves as config.yml
Directory target: ghcp <url> ./configs/ → saves as ./configs/filename.ext
No output specified: ghcp <url> → saves as filename.ext in current directory

Directory Downloads

Target directory: ghcp <url> ./output → extracts contents directly into ./output/
No output specified: ghcp <url> → extracts contents into current directory

Note: Directory contents are extracted directly into the target directory, not into a subdirectory named after the source directory.

Examples of Supported URLs

https://github.com/microsoft/vscode/blob/main/README.md
https://github.com/google/dart-basics/tree/master/lib
https://github.com/facebook/react/tree/main/packages
https://github.com/owner/repo/tree/feature-branch/src

Examples

Basic File Download

# Download docker-compose from a repository
ghcp https://github.com/open-webui/open-webui/blob/main/docker-compose.yaml

# Download with custom filename
ghcp https://github.com/open-webui/open-webui/blob/main/docker-compose.yaml compose.yaml

# Download to specific directory (keeps original filename)
ghcp https://github.com/google/dart-basics/blob/master/pubspec.yaml ./config/

Directory Downloads

# Download directory contents directly to current directory
ghcp https://github.com/google/dart-basics/tree/master/lib

# Download directory contents to specific folder
ghcp https://github.com/facebook/react/tree/main/src ./react-source

# Download documentation contents directly into docs folder
ghcp https://github.com/kubernetes/kubernetes/tree/master/docs ./k8s-docs

Private Repository Access

# Using environment variable (recommended for security)
export GITHUB_TOKEN=ghp_your_token_here
ghcp https://github.com/your-org/private-repo/blob/main/config/app.yaml

# Using command line flag (less secure, visible in process list)
ghcp https://github.com/your-org/private-repo/tree/main/src --token ghp_your_token_here

Advanced Examples

# Download specific branch content
ghcp https://github.com/flutter/flutter/tree/stable/packages/flutter/lib

# Download from a specific commit (use commit SHA as branch)
ghcp https://github.com/dart-lang/sdk/tree/abc123def456/pkg/analyzer

# Download nested directories
ghcp https://github.com/kubernetes/kubernetes/tree/master/cmd/kubectl/app

# Download to nested output directories
ghcp https://github.com/microsoft/vscode/tree/main/src/vs/editor ./editor-source

# Download large directories (will show progress for each file)
ghcp https://github.com/tensorflow/tensorflow/tree/master/tensorflow/python

Batch Downloads

# Download multiple files (using shell scripting)
for url in \
  "https://github.com/owner/repo/blob/main/file1.txt" \
  "https://github.com/owner/repo/blob/main/file2.md"; do
  ghcp "$url"
done

API Rate Limits

GitHub API has rate limits that affect ghcp:

Unauthenticated requests: 60 requests per hour
Authenticated requests: 5,000 requests per hour

Rate Limit Headers

The tool automatically handles rate limit information and provides clear error messages:

❌ GitHub API Error: API rate limit exceeded
Rate limit remaining: 0/60 (resets at 2024-01-01T12:00:00Z)

Avoiding Rate Limits

Use authentication: Authenticated requests have much higher limits (5,000 vs 60)
Batch operations: Download directories instead of individual files when possible
Be mindful of large directories: Each file in a directory requires a separate API call
Use caching: Avoid re-downloading the same content repeatedly

Rate Limit Best Practices

# Good: Downloads entire directory in one API call (plus one per file)
ghcp https://github.com/owner/repo/tree/main/src

# Less efficient: Multiple separate commands for files in same directory
ghcp https://github.com/owner/repo/blob/main/src/file1.dart
ghcp https://github.com/owner/repo/blob/main/src/file2.dart

Development

Building from Source

# Clone the repository
git clone https://github.com/aminnez/ghcp.git
cd ghcp

# Install dependencies
dart pub get

# Run in development mode
dart run bin/ghcp.dart --help

# Build standalone executable
dart compile exe bin/ghcp.dart -o ghcp

# Run tests
dart test

# Run static analysis
dart analyze

# Format code
dart format .

Troubleshooting

Common Issues

"Invalid GitHub URL"

Error: Invalid GitHub URL: Not a valid GitHub URL

Solutions:

Ensure URL starts with https://github.com/
Check URL format: https://github.com/owner/repo/tree/branch/path
Verify repository exists and is accessible

"API rate limit exceeded"

Error: GitHub API Error: API rate limit exceeded

Solutions:

Wait until rate limit resets (1 hour for unauthenticated)
Use authentication with GitHub token
Reduce request frequency

"Repository not found"

Error: GitHub API Error: Repository not found

Solutions:

Verify repository exists and is spelled correctly
Check if repository is private (requires authentication)
Ensure you have access to the repository

"No download URL available"

Error: Error: No download URL available for path/to/file

Solutions:

Verify the file exists in the repository
Check if it's a valid file (not a directory)
Ensure you have the correct branch name

Debug Mode

Enable verbose output for debugging:

# Set debug environment variable
export DEBUG=true
ghcp <url>

Network Issues

If you encounter network-related errors:

# Check network connectivity
ping github.com

# Test API accessibility
curl -I https://api.github.com

FAQ

Can I specify a custom filename for downloads?

Yes! For single file downloads, you can specify a custom filename:

# Download with custom filename
ghcp https://github.com/owner/repo/blob/main/config.yaml my-config.yml

# Download Docker Compose file with shorter name
ghcp https://github.com/owner/repo/blob/main/docker/docker-compose.yml dc.yml

Can I download from private repositories?

Yes! Use a GitHub Personal Access Token with appropriate permissions:

export GITHUB_TOKEN=your_token
ghcp https://github.com/your-org/private-repo/blob/main/file.txt

Does ghcp clone the entire repository?

No! ghcp only downloads the specific files or directories you request, making it much faster and more efficient than cloning.

Can I download multiple files at once?

Yes, by downloading a directory that contains multiple files:

ghcp https://github.com/owner/repo/tree/main/directory-with-multiple-files

What's the difference between blob and tree URLs?

blob URLs: Point to individual files
tree URLs: Point to directories (which may contain multiple files)

How do I know if a download succeeded?

ghcp provides visual feedback with progress spinners and completion messages. Successful downloads show:

Downloaded! filename.ext

Can I use ghcp in scripts?

Absolutely! ghcp is designed for scripting and automation:

#!/bin/bash
ghcp https://github.com/owner/repo/blob/main/config.yaml ./config
echo "Config downloaded successfully!"

Is there a size limit for downloads?

GitHub has specific file size limits:

Individual files: 100MB maximum (GitHub limitation)
Directory size: No specific limit, but consider:
- API rate limits (60 calls/hour without auth, 5,000 with auth)
- Each file requires a separate API call
- Large directories may take significant time
- Network and disk space constraints

How can I monitor download progress?

ghcp provides real-time visual feedback:

# Single file download
⏳ Downloading string_basics.dart...
✔ Downloaded! string_basics.dart

# Directory download  
✔ Downloading 3 of 11 files...
⠋ Downloading basics.dart...
⠋ Downloading comparable_basics.dart...
✔ Done! date_time_basics.dart
⠋ Downloading int_basics.dart...
✔ Done! 11 of 11 files.

Can I resume interrupted downloads?

Currently, ghcp does not support resume functionality. Interrupted downloads will need to be restarted. This is planned for a future release.

Does ghcp support GitHub Enterprise?

Currently, ghcp only supports GitHub.com. GitHub Enterprise support may be added in future versions based on user demand.

Contributing

Contributions are welcome! Here's how you can help:

Development Setup

Fork the repository

Clone your fork

git clone https://github.com/your-username/ghcp.git
cd ghcp

Install dependencies
```
dart pub get
```

Create a feature branch

git checkout -b feature/amazing-feature

Make your changes and ensure quality

# Run tests
dart test
   
# Run static analysis
dart analyze
   
# Format code
dart format .
   
# Check pub publish readiness
dart pub publish --dry-run

Commit your changes

git add .
git commit -m "Add amazing feature"
git push origin feature/amazing-feature

Submit a pull request

Local Testing

# Test the CLI locally
dart run bin/ghcp.dart <github-url> [output-dir]

# Build and test native executable
dart compile exe bin/ghcp.dart -o ghcp-test
./ghcp-test <github-url> [output-dir]

Code Style

This project follows Dart's official style guide with strict linting rules:

Type Safety: All public APIs must have explicit type annotations
Documentation: All public APIs should have comprehensive documentation
Error Handling: Use custom exceptions with descriptive messages
Testing: Maintain high test coverage for all new features

Linting Rules

The project uses strict analysis options defined in analysis_options.yaml:

# Check for linting issues
dart analyze

# Fix auto-fixable issues
dart fix --apply

Pull Request Guidelines

Small, focused changes: Keep PRs small and focused on a single feature/fix
Tests required: All new functionality must include tests
Documentation: Update README and inline docs as needed
Code quality: Ensure all lints pass and tests succeed
Backwards compatibility: Avoid breaking changes without discussion

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

Dart Team: For the amazing Dart programming language
GitHub: For providing excellent APIs and platform
Open Source Community: For inspiration and tools

GitHub Content Downloader (ghcp)

Features

Table of Contents

Installation

Option 1: Download Pre-built Binary (Recommended)

Option 2: Install from pub.flutter-io.cn

Option 3: Install from Source

Verify Installation

Quick Start

Usage

Command Syntax

Options

Authentication

Using Environment Variable (Recommended)

Using Command Line Flag

Configuration

Environment Variables

Creating a GitHub Token

Supported URL Formats

Single Files

Directories

Root Directory

Output Behavior

Single File Downloads

Directory Downloads

Examples of Supported URLs

Examples

Basic File Download

Directory Downloads

Private Repository Access

Advanced Examples

Batch Downloads

API Rate Limits

Rate Limit Headers

Avoiding Rate Limits

Rate Limit Best Practices

Development

Building from Source

Troubleshooting

Common Issues

"Invalid GitHub URL"

"API rate limit exceeded"

"Repository not found"

"No download URL available"

Debug Mode

Network Issues

FAQ

Can I specify a custom filename for downloads?

Can I download from private repositories?

Does ghcp clone the entire repository?

Can I download multiple files at once?

What's the difference between blob and tree URLs?

How do I know if a download succeeded?

Can I use ghcp in scripts?

Is there a size limit for downloads?

How can I monitor download progress?

Can I resume interrupted downloads?

Does ghcp support GitHub Enterprise?

Contributing

Development Setup

Local Testing

Code Style

Linting Rules

Pull Request Guidelines

License

Acknowledgments

Libraries

ghcp package