24  * struct rcu_tasks_percpu - Per-CPU component of definition for a Tasks-RCU-like mechanism.
32  * @rtp_blkd_tasks: List of tasks blocked as readers.
50  * struct rcu_tasks - Definition for a Tasks-RCU-like mechanism.
139 /* Track exiting tasks in order to allow them to be waited for. */
167 /* RCU tasks grace-period state for debugging. */
224 // Tasks RCU.
280 // Enqueue a callback for the specified flavor of Tasks RCU.
346 // Wait for all in-flight callbacks for the specified RCU Tasks flavor.
534 // RCU-tasks kthread that detects grace periods and invokes callbacks.
546 	 * one RCU-tasks grace period and then invokes the callbacks.  in rcu_tasks_kthread()
559 // Wait for a grace period for the specified flavor of Tasks RCU.
574 /* Spawn RCU-tasks grace-period kthread. */
588  * Print any non-default Tasks RCU settings.
604 	pr_info("\tTrampoline variant of Tasks RCU enabled.\n");  in rcu_tasks_bootup_oddness()
607 	pr_info("\tRude variant of Tasks RCU enabled.\n");  in rcu_tasks_bootup_oddness()
610 	pr_info("\tTracing variant of Tasks RCU enabled.\n");  in rcu_tasks_bootup_oddness()
617 /* Dump out rcutorture-relevant state common to all RCU-tasks flavors. */
649 // Shared code between task-list-scanning variants of Tasks RCU.
651 /* Wait for one RCU-tasks grace period. */
668 	 * There were callbacks, so we need to wait for an RCU-tasks  in rcu_tasks_wait_gp()
669 	 * grace period.  Start off by scanning the task list for tasks  in rcu_tasks_wait_gp()
670 	 * that are not already voluntarily blocked.  Mark these tasks  in rcu_tasks_wait_gp()
686 	 * tasks, removing any that are no longer holdouts.  When the list  in rcu_tasks_wait_gp()
771 //	Invoke synchronize_srcu() to ensure that all tasks that were
773 //	synchronize with this RCU Tasks grace period) have completed
781 //	t->on_rq and t->nvcsw transitions are seen by all CPUs and tasks
782 //	to have happened before the end of this RCU Tasks grace period.
830 	 * Wait for tasks that are in the process of exiting.  This  in rcu_tasks_postscan()
831 	 * does only part of the job, ensuring that all tasks that were  in rcu_tasks_postscan()
839 /* See if tasks are still holding out, complain if so. */
859 		pr_err("INFO: rcu_tasks detected stalls on tasks:\n");  in check_holdout_task()
871 /* Scan the holdout lists for tasks no longer holding out. */
883 /* Finish off the Tasks-RCU grace period. */
889 	 * reordering on other CPUs could cause their RCU-tasks read-side  in rcu_tasks_postgp()
899 	 * In addition, this synchronize_rcu() waits for exiting tasks  in rcu_tasks_postgp()
907 DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_wait_gp, call_rcu_tasks, "RCU Tasks");
921  * this primitive is intended to determine that all tasks have passed
934  * synchronize_rcu_tasks - wait until an rcu-tasks grace period has elapsed.
936  * Control will return to the caller some time after a full rcu-tasks
938  * executing rcu-tasks read-side critical sections have elapsed.  These
1019 // "Rude" variant of Tasks RCU, inspired by Steve Rostedt's trick of
1036 // Wait for one rude RCU-tasks grace period.
1048 		 "RCU Tasks Rude");
1062  * this primitive is intended to determine that all tasks have passed
1075  * synchronize_rcu_tasks_rude - wait for a rude rcu-tasks grace period
1077  * Control will return to the caller some time after a rude rcu-tasks
1079  * executing rcu-tasks read-side critical sections have elapsed.  These
1129 // Tracing variant of Tasks RCU.  This variant is designed to be used
1148 // flavors, rcu_preempt and rcu_sched.  The fact that RCU Tasks Trace
1158 //	Disables CPU hotplug, adds all currently executing tasks to the
1159 //	holdout list, then checks the state of all tasks that blocked
1160 //	or were preempted within their current RCU Tasks Trace read-side
1165 //	Invokes synchronize_rcu() to wait for late-stage exiting tasks
1175 // The exit_tasks_rcu_finish_trace() synchronizes with exiting tasks.
1205 		 "RCU Tasks Trace");
1242  * Also remove from the per-CPU list of blocked tasks.
1364 		// Check for "running" idle tasks on offline CPUs.  in trc_inspect_reader()
1477 /* Initialize for a new RCU-tasks-trace grace period. */
1506 	// Only after all running tasks have been accounted for is it  in rcu_tasks_trace_pregp_step()
1507 	// safe to take care of the tasks that have blocked within their  in rcu_tasks_trace_pregp_step()
1508 	// current RCU tasks trace read-side critical section.  in rcu_tasks_trace_pregp_step()
1536 	// Wait for late-stage exiting tasks to finish exiting.  in rcu_tasks_trace_postscan()
1539 	// Any tasks that exit after this point will set  in rcu_tasks_trace_postscan()
1543 /* Communicate task state back to the RCU tasks trace stall warning request. */
1562 /* Show the state of a task stalling the current RCU tasks trace GP. */
1570 		pr_err("INFO: rcu_tasks_trace detected stalls on tasks:\n");  in show_stalled_task_trace()
1593 /* List stalled IPIs for RCU tasks trace. */
1677  * The callback function will be invoked some time after a trace rcu-tasks
1679  * trace rcu-tasks read-side critical sections have completed. These
1693  * synchronize_rcu_tasks_trace - wait for a trace rcu-tasks grace period
1695  * Control will return to the caller some time after a trace rcu-tasks
1697  * trace rcu-tasks read-side critical sections have elapsed. These read-side
1711 …_held(&rcu_trace_lock_map), "Illegal synchronize_rcu_tasks_trace() in RCU Tasks Trace read-side cr…  in synchronize_rcu_tasks_trace()
1818 	pr_info("Running RCU-tasks wait API self tests\n");  in rcu_tasks_initiate_self_tests()