How to run background jobs?

Is there any library for the queueing mechanism?

What’s used by the most - Cron? But a task or rather script executed by Cron won’t access to the context of an application. Meaning, a task will have be an independent unit. Whereas I want is a library to use inside a project such that it’ll have access to everything.

Anything similar to Sidekiq exist in Rust?

ikornaselur,

I looked into this last week for recurring tasks in an actix-web project and ended up picking fang, but haven’t really battle tested it… But seems to do what I need. I’m using cron tasks with Postgres as the backend to run a recurring task every minute.

kevincox,
@kevincox@lemmy.ml avatar

For my project I just run them. I’m using async so it is just spawning a task in the background, but you can do the same with threads. Don’t underestimate the number of threads that you can run on a modern computer.

If you want some sort of throttling you can stuff tasks into a queue and just run N background threads pulling them off an processing them.

If you need durability then you start to have more trouble. This is where I would start looking at a library. IDK if there are any libraries that handle logging, retries and similar, but if not you can probably get the basics down pretty easily.

nothingness,

For my project I just run them.

How would you “just run” a task every 30 minutes? Every 5 hours? Once a day?

kevincox,
@kevincox@lemmy.ml avatar 

<span style="color:#323232;">std::thread::spawn(|| {
</span><span style="color:#323232;">    loop {
</span><span style="color:#323232;">        std::thread::sleep(std::time::Duration::from_secs(30*60));
</span><span style="color:#323232;">        do_job();
</span><span style="color:#323232;">    }
</span><span style="color:#323232;">});
</span>

Works pretty well. Maybe add a bit of code to crash the whole process on panic or some other logging. Wastes a few KiB of memory per loop but probably not a major issue. Doing this with async will waste only the tiniest amount of memory.

vest,
@vest@mastodon.online avatar

@kevincox How do you stop the job? Do you use channels like in Go?

kevincox,
@kevincox@lemmy.ml avatar

It depends. Sometimes you can just put an exit call at the end of main to kill the thread. If you want to attempt graceful shutdown then usually I just use a boolean shutdown flag. Then the loop becomes while !shutdown.get() {

nothingness,

what if one of the calls crashes? how would you re-run it?

kevincox,
@kevincox@lemmy.ml avatar

Best option is probably to add a wrapper around the thread that re-spawns it. But you can also just catch panics in the loop.

nothingness, (edited )

How would you re-run it multiple times then? An internal should be progressively greater. How would you terminate it if it continues to produce an exception?

smorks,
@smorks@lemmy.ca avatar

lemmy uses clockwerk i believe, there’s a few out there for rust.

SuddenlyBlowGreen, 

<span style="color:#323232;">let mut scheduler = Scheduler::with_tz(chrono::Utc);
</span><span style="color:#323232;">
</span><span style="color:#323232;">scheduler.every(10.minutes()).plus(30.seconds()).run(|| println!("Periodic task"));
</span><span style="color:#323232;">
</span><span style="color:#323232;">scheduler.every(1.day()).at("3:20 pm").run(|| println!("Daily task")); 
</span><span style="color:#323232;">
</span><span style="color:#323232;">scheduler.every(Tuesday).at("14:20:17").and_every(Thursday).at("15:00").run(|| println!("Biweekly task")); 
</span>

Damn, that a really ingenious and intuitive use of the builder pattern.

Kudos to the devs!

nothingness,

Scheduler

what library is it from?

SuddenlyBlowGreen,

The clokwerk crate.

peterprototypes,
@peterprototypes@mastodon.social avatar

@SuddenlyBlowGreen @nothingness I recently needed that, but fuzzy. Like I want a task ran anywhere between 1PM and 2PM. Can this be accomplished with the clockwerk crate?

SuddenlyBlowGreen,

You could schedule a task at 1 PM, then generate a random number between 0-60 inside that task, wait that many minutes, then launch the actual task.

vaalla,

Can you use async in your project?

If Yes, you can spawn a task that will listen on a channel. If you need to run them in parallel probably you can find a mpmc channel.

If you need for them to run at a specific time, spawn task ,tokio::time::sleep , run job, loop.

Don’t know any crate just for this.

nothingness,

If you need for them to run at a specific time, spawn task ,tokio::time::sleep , run job, loop.

How would you do it every 30 minutes? Every 5 hours? Once a day?

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • random
  • uselessserver093
  • Food
  • aaaaaaacccccccce
  • [email protected]
  • test
  • CafeMeta
  • testmag
  • MUD
  • RhythmGameZone
  • RSS
  • dabs
  • Socialism
  • KbinCafe
  • TheResearchGuardian
  • oklahoma
  • feritale
  • SuperSentai
  • KamenRider
  • All magazines
    •