As generative synthetic intelligence (AI) inference turns into more and more essential for companies, prospects are searching for methods to scale their generative AI operations or combine generative AI fashions into present workflows. Mannequin optimization has emerged as a vital step, permitting organizations to steadiness cost-effectiveness and responsiveness, enhancing productiveness. Nonetheless, price-performance necessities range extensively throughout use instances. For chat purposes, minimizing latency is essential to supply an interactive expertise, whereas real-time purposes like suggestions require maximizing throughput. Navigating these trade-offs poses a major problem to quickly undertake generative AI, since you should rigorously choose and consider totally different optimization strategies.

To beat these challenges, we’re excited to introduce the inference optimization toolkit, a totally managed mannequin optimization function in Amazon SageMaker. This new function delivers as much as ~2x greater throughput whereas decreasing prices by as much as ~50% for generative AI fashions equivalent to Llama 3, Mistral, and Mixtral fashions. For instance, with a Llama 3-70B mannequin, you possibly can obtain as much as ~2400 tokens/sec on a ml.p5.48xlarge occasion v/s ~1200 tokens/sec beforehand with none optimization.

This inference optimization toolkit makes use of the most recent generative AI mannequin optimization strategies equivalent to compilation, quantization, and speculative decoding that will help you scale back the time to optimize generative AI fashions from months to hours, whereas reaching the very best price-performance to your use case. For compilation, the toolkit makes use of the Neuron Compiler to optimize the mannequin’s computational graph for particular {hardware}, equivalent to AWS Inferentia, enabling sooner runtimes and decreased useful resource utilization. For quantization, the toolkit makes use of Activation-aware Weight Quantization (AWQ) to effectively shrink the mannequin dimension and reminiscence footprint whereas preserving high quality. For speculative decoding, the toolkit employs a sooner draft mannequin to foretell candidate outputs in parallel, enhancing inference velocity for longer textual content era duties. To study extra about every approach, seek advice from Optimize mannequin inference with Amazon SageMaker. For extra particulars and benchmark outcomes for widespread open supply fashions, see Obtain as much as ~2x greater throughput whereas decreasing prices by as much as ~50% for generative AI inference on Amazon SageMaker with the brand new inference optimization toolkit – Half 1.

On this submit, we exhibit the best way to get began with the inference optimization toolkit for supported fashions in Amazon SageMaker JumpStart and the Amazon SageMaker Python SDK. SageMaker JumpStart is a totally managed mannequin hub that permits you to discover, fine-tune, and deploy widespread open supply fashions with only a few clicks. You should utilize pre-optimized fashions or create your individual {custom} optimizations. Alternatively, you possibly can accomplish this utilizing the SageMaker Python SDK, as proven within the following pocket book. For the total record of supported fashions, seek advice from Optimize mannequin inference with Amazon SageMaker.

Utilizing pre-optimized fashions in SageMaker JumpStart

The inference optimization toolkit offers pre-optimized fashions which were optimized for best-in-class cost-performance at scale, with none affect to accuracy. You may select the configuration based mostly on the latency and throughput necessities of your use case and deploy in a single click on.

Taking the Meta-Llama-3-8b mannequin in SageMaker JumpStart for instance, you possibly can select Deploy from the mannequin web page. Within the deployment configuration, you possibly can broaden the mannequin configuration choices, choose the variety of concurrent customers, and deploy the optimized mannequin.

Deploying a pre-optimized mannequin with the SageMaker Python SDK

It’s also possible to deploy a pre-optimized generative AI mannequin utilizing the SageMaker Python SDK in only a few traces of code. Within the following code, we arrange a ModelBuilder class for the SageMaker JumpStart mannequin. ModelBuilder is a category within the SageMaker Python SDK that gives fine-grained management over numerous deployment facets, equivalent to occasion varieties, community isolation, and useful resource allocation. You should utilize it to create a deployable mannequin occasion, changing framework fashions (like XGBoost or PyTorch) or Inference Specs into SageMaker-compatible fashions for deployment. Seek advice from Create a mannequin in Amazon SageMaker with ModelBuilder for extra particulars.

sample_input = {
    "inputs": "Hi there, I am a language mannequin,",
    "parameters": {"max_new_tokens":128, "do_sample":True}
}

sample_output = [
    {
        "generated_text": "Hello, I'm a language model, and I'm here to help you with your English."
    }
]
schema_builder = SchemaBuilder(sample_input, sample_output)

builder = ModelBuilder(
    mannequin="meta-textgeneration-llama-3-8b", # JumpStart mannequin ID
    schema_builder=schema_builder,
    role_arn=role_arn,
)

Record the accessible pre-benchmarked configurations with the next code:

builder.display_benchmark_metrics()

Select the suitable instance_type and config_name from the record based mostly in your concurrent customers, latency, and throughput necessities. Within the previous desk, you possibly can see the latency and throughput throughout totally different concurrency ranges for the given occasion sort and config title. If config title is lmi-optimized, meaning the configuration is pre-optimized by SageMaker. Then you possibly can name .construct() to run the optimization job. When the job is full, you possibly can deploy to an endpoint and take a look at the mannequin predictions. See the next code:

# set deployment config with pre-configured optimization
bulder.set_deployment_config(
    instance_type="ml.g5.12xlarge", 
    config_name="lmi-optimized"
)

# construct the deployable mannequin
mannequin = builder.construct()

# deploy the mannequin to a SageMaker endpoint
predictor = mannequin.deploy(accept_eula=True)

# use pattern enter payload to check the deployed endpoint
predictor.predict(sample_input)

Utilizing the inference optimization toolkit to create {custom} optimizations

Along with making a pre-optimized mannequin, you possibly can create {custom} optimizations based mostly on the occasion sort you select. The next desk offers a full record of accessible combos. Within the following sections, we discover compilation on AWS Inferentia first, after which attempt the opposite optimization strategies for GPU cases.

Compilation from SageMaker JumpStart

For compilation, you possibly can choose the identical Meta-Llama-3-8b mannequin from SageMaker JumpStart and select Optimize on the mannequin web page. Within the optimization configuration web page, you possibly can select ml.inf2.8xlarge to your occasion sort. Then present an output Amazon Easy Storage Service (Amazon S3) location for the optimized artifacts. For big fashions like Llama 2 70B, for instance, the compilation job can take greater than an hour. Subsequently, we suggest utilizing the inference optimization toolkit to carry out ahead-of-time compilation. That means, you solely must compile one time.

Compilation utilizing the SageMaker Python SDK

For the SageMaker Python SDK, you possibly can configure the compilation by altering the atmosphere variables within the .optimize() perform. For extra particulars on compilation_config, seek advice from LMI NeuronX ahead-of-time compilation of fashions tutorial.

compiled_model = builder.optimize(
    instance_type="ml.inf2.8xlarge",
    accept_eula=True,
    compilation_config={
        "OverrideEnvironment": {
            "OPTION_TENSOR_PARALLEL_DEGREE": "2",
            "OPTION_N_POSITIONS": "2048",
            "OPTION_DTYPE": "fp16",
            "OPTION_ROLLING_BATCH": "auto",
            "OPTION_MAX_ROLLING_BATCH_SIZE": "4",
            "OPTION_NEURON_OPTIMIZE_LEVEL": "2",
        }
   },
   output_path=f"s3://{output_bucket_name}/compiled/"
)

# deploy the compiled mannequin to a SageMaker endpoint
predictor = compiled_model.deploy(accept_eula=True)

# use pattern enter payload to check the deployed endpoint
predictor.predict(sample_input)

Quantization and speculative decoding from SageMaker JumpStart

For optimizing fashions on GPU, ml.g5.12xlarge is the default deployment occasion sort for Llama-3-8b. You may select quantization, speculative decoding, or each as optimization choices. Quantization makes use of AWQ to scale back the mannequin’s weights to low-bit (INT4) representations. Lastly, you possibly can present an output S3 URL to retailer the optimized artifacts.

With speculative decoding, you possibly can enhance latency and throughput by both utilizing the SageMaker supplied draft mannequin or bringing your individual draft mannequin from the general public Hugging Face mannequin hub or add from your individual S3 bucket.

After the optimization job is full, you possibly can deploy the mannequin or run additional analysis jobs on the optimized mannequin. On the SageMaker Studio UI, you possibly can select to make use of the default pattern datasets or present your individual utilizing an S3 URI. On the time of writing, the consider efficiency choice is just accessible by means of the Amazon SageMaker Studio UI.

Quantization and speculative decoding utilizing the SageMaker Python SDK

The next is the SageMaker Python SDK code snippet for quantization. You simply want to supply the quantization_config attribute within the .optimize() perform.

optimized_model = builder.optimize(
    instance_type="ml.g5.12xlarge",
    accept_eula=True,
    quantization_config={
        "OverrideEnvironment": {
            "OPTION_QUANTIZE": "awq"
        }
    },
    output_path=f"s3://{output_bucket_name}/quantized/"
)

# deploy the optimized mannequin to a SageMaker endpoint
predictor = optimized_model.deploy(accept_eula=True)

# use pattern enter payload to check the deployed endpoint
predictor.predict(sample_input)

For speculative decoding, you possibly can change to a speculative_decoding_config attribute and configure SageMaker or a {custom} mannequin. Chances are you’ll want to regulate the GPU utilization based mostly on the sizes of the draft and goal fashions to suit them each on the occasion for inference.

optimized_model = builder.optimize(
    instance_type="ml.g5.12xlarge",
    accept_eula=True,
    speculative_decoding_config={
        "ModelProvider": "sagemaker"
    }
    # speculative_decoding_config={
        # "ModelProvider": "{custom}",
        # use S3 URI or HuggingFace mannequin ID for {custom} draft mannequin
        # notice: utilizing HuggingFace mannequin ID as draft mannequin requires HF_TOKEN in atmosphere variables
        # "ModelSource": "s3://custom-bucket/draft-model", 
    # }
)

# deploy the optimized mannequin to a SageMaker endpoint
predictor = optimized_model.deploy(accept_eula=True)

# use pattern enter payload to check the deployed endpoint
predictor.predict(sample_input)

Conclusion

Optimizing generative AI fashions for inference efficiency is essential for delivering cost-effective and responsive generative AI options. With the launch of the inference optimization toolkit, now you can optimize your generative AI fashions, utilizing the most recent strategies equivalent to speculative decoding, compilation, and quantization to attain as much as ~2x greater throughput and scale back prices by as much as ~50%. This helps you obtain the optimum price-performance steadiness to your particular use instances with only a few clicks in SageMaker JumpStart or a number of traces of code utilizing the SageMaker Python SDK. The inference optimization toolkit considerably simplifies the mannequin optimization course of, enabling your corporation to speed up generative AI adoption and unlock extra alternatives to drive higher enterprise outcomes.

To study extra, seek advice from Optimize mannequin inference with Amazon SageMaker and Obtain as much as ~2x greater throughput whereas decreasing prices by as much as ~50% for generative AI inference on Amazon SageMaker with the brand new inference optimization toolkit – Half 1.

Concerning the Authors

James Wu is a Senior AI/ML Specialist Options Architect
Saurabh Trikande is a Senior Product Supervisor
Rishabh Ray Chaudhury is a Senior Product Supervisor
Kumara Swami Borra is a Entrance Finish Engineer
Alwin (Qiyun) Zhao is a Senior Software program Growth Engineer
Qing Lan is a Senior SDE