{"id":2261,"date":"2026-09-16T14:44:31","date_gmt":"2026-09-16T12:44:31","guid":{"rendered":"https:\/\/www.ideas.edu.pl\/?post_type=publikacje&#038;p=2261"},"modified":"2026-09-16T14:44:31","modified_gmt":"2026-09-16T12:44:31","slug":"sacn-soft-actor-critic-with-n-step-returns","status":"publish","type":"publikacje","link":"https:\/\/www.ideas.edu.pl\/en\/publikacje\/sacn-soft-actor-critic-with-n-step-returns\/","title":{"rendered":"SACn: Soft Actor-Critic with n-Step Returns"},"content":{"rendered":"<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">Soft Actor-Critic (SAC) is widely used in practical applications and is now one of the most relevant off-policy online model-free reinforcement learning (RL) methods. The technique of n-step returns is known to increase the convergence speed of RL algorithms compared to their 1-step returns-based versions. However, SAC is notoriously difficult to combine with n-step returns, since their usual combination introduces bias in off-policy algorithms due to changes in the action distribution. While this problem is solved by importance sampling\u2014a method for estimating the expected values of one distribution using samples from another distribution\u2014importance sampling can lead to numerical instability. In this work, we combine SAC with n-step returns in a way that overcomes this issue. We present an approach to applying numerically stable importance sampling with simplified hyperparameter selection. Furthermore, we analyze the entropy estimation approach of Soft Actor-Critic within the context of the n-step maximum entropy framework and formulate the \u03c4-sampled entropy estimation to reduce the variance of the learning target. Finally, we formulate the Soft Actor-Critic with n-step returns (SACn) algorithm, which we experimentally validate on MuJoCo simulated environments.<\/p>\n<\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">Authors: Jakub \u0141yskawa, Jakub Lewandowski, Pawe\u0142 Wawrzy\u0144ski<\/p>\n\n\n\n<div style=\"height:64px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<div class=\"wp-block-group is-layout-grid wp-container-core-group-is-layout-5ab56a3c wp-block-group-is-layout-grid\"><\/div>\n\n\n\n<div class=\"wp-block-group is-layout-grid wp-container-core-group-is-layout-5ab56a3c wp-block-group-is-layout-grid\"><\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>","protected":false},"template":"","nazwa-konferencji":[49],"rodzaj-publikacji":[13],"rok-publikacji":[27],"class_list":["post-2261","publikacje","type-publikacje","status-publish","hentry","nazwa-konferencji-konferencja-icaart","rodzaj-publikacji-artykul-konferencyjny","rok-publikacji-27"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>SACn: Soft Actor-Critic with n-step Returns &#8226; IDEAS Instytut Badawczy<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.ideas.edu.pl\/en\/publikacje\/sacn-soft-actor-critic-with-n-step-returns\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"SACn: Soft Actor-Critic with n-step Returns &#8226; IDEAS Instytut Badawczy\" \/>\n<meta property=\"og:description\" content=\"Soft Actor-Critic (SAC) is widely used in practical applications and is now one of the most relevant off-policy online model-free reinforcement learning (RL) methods. The technique of n-step returns is known to increase the convergence speed of RL algorithms compared to their 1-step returns-based versions. However, SAC is notoriously difficult to combine with n-step returns, [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.ideas.edu.pl\/en\/publikacje\/sacn-soft-actor-critic-with-n-step-returns\/\" \/>\n<meta property=\"og:site_name\" content=\"IDEAS Instytut Badawczy\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.ideas.edu.pl\/wp-content\/uploads\/feature-image-home.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"1800\" \/>\n\t<meta property=\"og:image:height\" content=\"945\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.ideas.edu.pl\\\/publikacje\\\/sacn-soft-actor-critic-with-n-step-returns\\\/\",\"url\":\"https:\\\/\\\/www.ideas.edu.pl\\\/publikacje\\\/sacn-soft-actor-critic-with-n-step-returns\\\/\",\"name\":\"SACn: Soft Actor-Critic with n-step Returns &#8226; IDEAS Instytut Badawczy\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.ideas.edu.pl\\\/#website\"},\"datePublished\":\"2026-09-16T12:44:31+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.ideas.edu.pl\\\/publikacje\\\/sacn-soft-actor-critic-with-n-step-returns\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.ideas.edu.pl\\\/publikacje\\\/sacn-soft-actor-critic-with-n-step-returns\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.ideas.edu.pl\\\/publikacje\\\/sacn-soft-actor-critic-with-n-step-returns\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Strona g\u0142\u00f3wna\",\"item\":\"https:\\\/\\\/www.ideas.edu.pl\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Publikacje\",\"item\":\"https:\\\/\\\/www.ideas.edu.pl\\\/publikacje\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"SACn: Soft Actor-Critic with n-step Returns\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.ideas.edu.pl\\\/#website\",\"url\":\"https:\\\/\\\/www.ideas.edu.pl\\\/\",\"name\":\"IDEAS Instytut Badawczy\",\"description\":\"Pa\u0144stwowa jednostka badawczo-naukowa\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.ideas.edu.pl\\\/#organization\"},\"alternateName\":\"IDEAS\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.ideas.edu.pl\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.ideas.edu.pl\\\/#organization\",\"name\":\"IDEAS Instytut Badawczy\",\"url\":\"https:\\\/\\\/www.ideas.edu.pl\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.ideas.edu.pl\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.ideas.edu.pl\\\/wp-content\\\/uploads\\\/e90241d6e73025d0d829abc28d67cb84.svg\",\"contentUrl\":\"https:\\\/\\\/www.ideas.edu.pl\\\/wp-content\\\/uploads\\\/e90241d6e73025d0d829abc28d67cb84.svg\",\"width\":152,\"height\":43,\"caption\":\"IDEAS Instytut Badawczy\"},\"image\":{\"@id\":\"https:\\\/\\\/www.ideas.edu.pl\\\/#\\\/schema\\\/logo\\\/image\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"SACn: Soft Actor-Critic with n-step Returns &#8226; IDEAS Instytut Badawczy","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.ideas.edu.pl\/en\/publikacje\/sacn-soft-actor-critic-with-n-step-returns\/","og_locale":"en_US","og_type":"article","og_title":"SACn: Soft Actor-Critic with n-step Returns &#8226; IDEAS Instytut Badawczy","og_description":"Soft Actor-Critic (SAC) is widely used in practical applications and is now one of the most relevant off-policy online model-free reinforcement learning (RL) methods. The technique of n-step returns is known to increase the convergence speed of RL algorithms compared to their 1-step returns-based versions. However, SAC is notoriously difficult to combine with n-step returns, [&hellip;]","og_url":"https:\/\/www.ideas.edu.pl\/en\/publikacje\/sacn-soft-actor-critic-with-n-step-returns\/","og_site_name":"IDEAS Instytut Badawczy","og_image":[{"width":1800,"height":945,"url":"https:\/\/www.ideas.edu.pl\/wp-content\/uploads\/feature-image-home.webp","type":"image\/webp"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.ideas.edu.pl\/publikacje\/sacn-soft-actor-critic-with-n-step-returns\/","url":"https:\/\/www.ideas.edu.pl\/publikacje\/sacn-soft-actor-critic-with-n-step-returns\/","name":"SACn: Soft Actor-Critic with n-step Returns &#8226; IDEAS Instytut Badawczy","isPartOf":{"@id":"https:\/\/www.ideas.edu.pl\/#website"},"datePublished":"2026-09-16T12:44:31+00:00","breadcrumb":{"@id":"https:\/\/www.ideas.edu.pl\/publikacje\/sacn-soft-actor-critic-with-n-step-returns\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.ideas.edu.pl\/publikacje\/sacn-soft-actor-critic-with-n-step-returns\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.ideas.edu.pl\/publikacje\/sacn-soft-actor-critic-with-n-step-returns\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Strona g\u0142\u00f3wna","item":"https:\/\/www.ideas.edu.pl\/"},{"@type":"ListItem","position":2,"name":"Publikacje","item":"https:\/\/www.ideas.edu.pl\/publikacje\/"},{"@type":"ListItem","position":3,"name":"SACn: Soft Actor-Critic with n-step Returns"}]},{"@type":"WebSite","@id":"https:\/\/www.ideas.edu.pl\/#website","url":"https:\/\/www.ideas.edu.pl\/","name":"IDEAS Research Institute","description":"State research and scientific unit","publisher":{"@id":"https:\/\/www.ideas.edu.pl\/#organization"},"alternateName":"IDEAS","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.ideas.edu.pl\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.ideas.edu.pl\/#organization","name":"IDEAS Research Institute","url":"https:\/\/www.ideas.edu.pl\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.ideas.edu.pl\/#\/schema\/logo\/image\/","url":"https:\/\/www.ideas.edu.pl\/wp-content\/uploads\/e90241d6e73025d0d829abc28d67cb84.svg","contentUrl":"https:\/\/www.ideas.edu.pl\/wp-content\/uploads\/e90241d6e73025d0d829abc28d67cb84.svg","width":152,"height":43,"caption":"IDEAS Instytut Badawczy"},"image":{"@id":"https:\/\/www.ideas.edu.pl\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/www.ideas.edu.pl\/en\/wp-json\/wp\/v2\/publikacje\/2261","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ideas.edu.pl\/en\/wp-json\/wp\/v2\/publikacje"}],"about":[{"href":"https:\/\/www.ideas.edu.pl\/en\/wp-json\/wp\/v2\/types\/publikacje"}],"wp:attachment":[{"href":"https:\/\/www.ideas.edu.pl\/en\/wp-json\/wp\/v2\/media?parent=2261"}],"wp:term":[{"taxonomy":"nazwa-konferencji","embeddable":true,"href":"https:\/\/www.ideas.edu.pl\/en\/wp-json\/wp\/v2\/nazwa-konferencji?post=2261"},{"taxonomy":"rodzaj-publikacji","embeddable":true,"href":"https:\/\/www.ideas.edu.pl\/en\/wp-json\/wp\/v2\/rodzaj-publikacji?post=2261"},{"taxonomy":"rok-publikacji","embeddable":true,"href":"https:\/\/www.ideas.edu.pl\/en\/wp-json\/wp\/v2\/rok-publikacji?post=2261"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}