Charith Asalanka (L) and Chamika Karunaratne (R) leaves the field after defeating Australia in Colombo on Sunday. AP
Set a tricky 292 to win after a Travis Head 70, the hosts won comfortably with nine balls to spare in Colombo, in their second straight victory in the five-match series.
Maxwell bowled Niroshan Dickwella for 25 but Nissanka and Kusal Mendis ground down the injury-depleted Australian attack of Josh Hazlewood, Jhye Richardson and Matthew Kuhnemann.
Chamika Karunaratne (C) celebrates with his teammate Charith Asalanka after their win. AFP
Mendis retired hurt unbeaten on 87 and was replaced by Dhananjaya de Silva who made 25, but Nissanka clung on until almost the end, caught off Richardson for 137 off 147 balls.
Richardson then bowled captain Dasun Shanaka for a duck, but Charith AsaLanka — with an ice-cold six in the penultimate over off Hazlewood — and Chamika Karunaratne saw Sri Lanka home.
Earlier Australia braved a spin onslaught by Sri Lanka to reach 291-6 in their 50 overs after winning the toss and opting to bat first.
Travis Head plays a shot during the third ODI match at R. Premadasa International Cricket Stadium. AFP
Opener David Warner went for nine in the third over, top-edging a Dushmantha Chameera bouncer to Maheesh Theekshana at fine leg.
Back from injury, Mitchell Marsh struggled to get going and departed for 10 off 23 deliveries, caught at cover by Mendis after a slower delivery by Dunith Wellalage.
Marnus Labuschagne came in at number four ahead of Head and he and Aaron Finch added 69 before Labuschagne was stumped by Niroshan Dickwella off Jeffrey Vandersay for 29.
Two overs later, Vandersay bagged Finch too, with the Australian captain edging the spinner to a diving de Silva at slip having made 62 off 85.
Cricket fans cheer in support of Sri Lanka during the ODI match. AFP
Head and Alex Carey added 72 runs before de Silva trapped Carey leg-before on the front pad trying to sweep, having made a quick-fire 49 off 52 deliveries.
Maxwell did some damage, smashing 33 off 18 before giving Vandersay his third wicket, miscuing the spinner into the hands of Wellalage scurrying back from midwicket.
But Head stayed firm, reaching his half-century and smashing Wellalage for three sixes in the penultimate over and ending 70 not out off 65 deliveries, with Cameron Green unbeaten on 15. Vandersay ended with figures of 3 for 49.
Already missing Marcus Stoinis, Kane Richardson and Mitchell Starc, Australia left out Steve Smith after the batsman injured his left quad batting in Thursday's 26-run defeat in Pallekele.
Glenn Maxwell plays a shot during the third ODI. AFP
Marsh, however, returned to the side, as did Green and Richardson, while Pat Cummins and Mitchell Swepson were left out. Spinner Kuhnemann was again in for Ashton Agar.
The hosts were once again without key all-rounder Wanindu Hasaranga, who injured his groin in the first match while Dickwella replaced Danushka Gunathilaka.
Australia: Aaron Finch (captain), David Warner, Mitchell Marsh, Marnus Labuschagne, Alex Carey, Travis Head, Glenn Maxwell, Matthew Kuhnemann, Jhye Richardson, Cameron Green, Josh Hazlewood.
Sri Lanka: Niroshan Dickwella, Pathum Nissanka, Charith Asalanka, Kusal Mendis, Dhananjaya de Silva, Dasun Shanaka (captain), Dunith Wellalage, Jeffrey Vandersay, Chamika Karunaratne, Dushmantha Chameera, Maheesh Theekshana.