WEBVTT 1 00:00:01.230 --> 00:00:02.970 - Thanks Pete. Hi everyone. 2 00:00:02.970 --> 00:00:04.680 Welcome. Good afternoon. 3 00:00:04.680 --> 00:00:07.719 Or good morning if you're on the West Coast. 4 00:00:07.719 --> 00:00:09.030 Thanks for joining us. 5 00:00:09.030 --> 00:00:14.030 This is the first of a two- part series that focuses 6 00:00:14.200 --> 00:00:17.640 on measurement and sampling for CFSR Round 4. 7 00:00:17.640 --> 00:00:20.730 Today's session, we're going to focus on measurement 8 00:00:20.730 --> 00:00:24.750 and sampling for the CFSR portion of the CFSR cycle. 9 00:00:24.750 --> 00:00:27.073 And next week, we'll turn to the PIP. 10 00:00:28.200 --> 00:00:30.360 So this is the first of a two-part series, 11 00:00:30.360 --> 00:00:33.010 but the two-part series is of course itself part 12 00:00:33.010 --> 00:00:36.040 of a larger series of national meetings, 13 00:00:36.040 --> 00:00:38.883 where we talk about activities associated with the CFSR. 14 00:00:40.100 --> 00:00:43.643 So today you'll be hearing from me. 15 00:00:43.643 --> 00:00:45.130 My name is Jennifer Haight. 16 00:00:45.130 --> 00:00:46.530 I'm the director of the Division 17 00:00:46.530 --> 00:00:49.050 of Performance Measurement and Improvement. 18 00:00:49.050 --> 00:00:52.280 In that division is the data team, 19 00:00:52.280 --> 00:00:55.070 which is also sometimes called DART, which stands 20 00:00:55.070 --> 00:00:58.140 for Data Analytics and Reporting Team. 21 00:00:58.140 --> 00:00:59.980 Roger Stanton, who's part of that team, 22 00:00:59.980 --> 00:01:02.120 will be doing some of the presenting 23 00:01:02.120 --> 00:01:05.670 on the statewide data indicators. 24 00:01:05.670 --> 00:01:09.610 And he'll be joined by Sylvia Kim, who is a member 25 00:01:09.610 --> 00:01:13.553 of the Child and Family Services or the CFSR team. 26 00:01:14.540 --> 00:01:17.400 And she also serves on the MASC or the Measurement 27 00:01:17.400 --> 00:01:20.410 and Sampling Committee, which is the committee 28 00:01:20.410 --> 00:01:24.490 responsible for providing internal and external advice 29 00:01:24.490 --> 00:01:29.490 to states and jurisdictions about the tactics we use 30 00:01:30.160 --> 00:01:33.870 to measure performance for both the CFSR and the PIPs. 31 00:01:33.870 --> 00:01:37.760 And Tammy White, who is a member of the data team 32 00:01:37.760 --> 00:01:39.230 and sits on MASC. 33 00:01:39.230 --> 00:01:40.453 So she sort of has, 34 00:01:41.470 --> 00:01:44.120 as everyone on this call with us who are squarely 35 00:01:44.120 --> 00:01:47.663 in the measurement and in the CFSR camp. 36 00:01:48.860 --> 00:01:52.840 So today our focus is, as I said, 37 00:01:52.840 --> 00:01:55.930 specifically on measurement and sampling as it relates 38 00:01:55.930 --> 00:01:57.510 to the CFSR. 39 00:01:57.510 --> 00:01:59.100 We're going to start with a focus 40 00:01:59.100 --> 00:02:01.296 on the statewide data indicators. 41 00:02:01.296 --> 00:02:04.770 We'll provide a quick overview 42 00:02:04.770 --> 00:02:08.200 about what those Round 4 indicators are 43 00:02:08.200 --> 00:02:12.250 and some updates that we've made to the way 44 00:02:12.250 --> 00:02:16.870 we calculate them or generate them, not very many updates, 45 00:02:16.870 --> 00:02:20.490 just as a hint of what's to come. 46 00:02:20.490 --> 00:02:22.060 We'll talk about the way, 47 00:02:22.060 --> 00:02:25.433 the method for determining the substantial conformity. 48 00:02:26.564 --> 00:02:30.180 What we can tell you about the impact of AFCARS 2020, 49 00:02:30.180 --> 00:02:33.700 which as many of you know is the data collection 50 00:02:33.700 --> 00:02:37.360 we are transitioning to during Round 4, we'll provide 51 00:02:37.360 --> 00:02:40.090 some resources and there'll be time for questions. 52 00:02:40.090 --> 00:02:43.230 We'll then turn to the CFSR case review framework 53 00:02:43.230 --> 00:02:47.900 and focus on the population that's to be selected 54 00:02:47.900 --> 00:02:49.840 for CFSR case reviews, 55 00:02:49.840 --> 00:02:52.260 and some details about selection sampling, 56 00:02:52.260 --> 00:02:55.610 and how to consider the period under review. 57 00:02:55.610 --> 00:02:59.130 We'll talk a little bit about case elimination, and again, 58 00:02:59.130 --> 00:03:01.203 have resources and questions. 59 00:03:04.771 --> 00:03:06.950 I'm going to start first and provide 60 00:03:06.950 --> 00:03:11.300 an overview of the case that CFSR statewide 61 00:03:11.300 --> 00:03:15.230 data indicators, and then turn 62 00:03:15.230 --> 00:03:17.850 it over to Roger for a more in-depth discussion 63 00:03:17.850 --> 00:03:19.200 about those indicators. 64 00:03:19.200 --> 00:03:23.080 And the one thing I wanted to say before I get started 65 00:03:23.080 --> 00:03:25.780 is that a lot of this information is located 66 00:03:25.780 --> 00:03:28.420 in Technical Bulletin #13. 67 00:03:28.420 --> 00:03:30.520 We wanted to take this opportunity to review 68 00:03:30.520 --> 00:03:33.440 it more slowly with you, to take questions. 69 00:03:33.440 --> 00:03:35.720 So please feel free to type your questions 70 00:03:35.720 --> 00:03:38.490 in the questions-and-answers area. 71 00:03:38.490 --> 00:03:41.090 We will be monitoring them and we will be addressing 72 00:03:42.210 --> 00:03:44.610 all of them to the extent possible on this call. 73 00:03:47.250 --> 00:03:48.083 - [Elizabeth] Jen, for this list, 74 00:03:48.083 --> 00:03:50.240 there's quite a few people saying 75 00:03:50.240 --> 00:03:53.700 that they're having audio challenges joining 76 00:03:53.700 --> 00:03:54.810 through the device speaker. 77 00:03:54.810 --> 00:03:59.810 So maybe, could we just encourage all the participants 78 00:03:59.820 --> 00:04:02.520 who are having difficulty to join through their phone? 79 00:04:03.490 --> 00:04:06.310 - [Roger] I, yeah, I, Jennifer, I think you might just want 80 00:04:06.310 --> 00:04:08.750 to speak a little bit, if possible, 81 00:04:08.750 --> 00:04:09.700 speak a little bit louder, 82 00:04:09.700 --> 00:04:12.420 maybe a little closer to the microphone. 83 00:04:12.420 --> 00:04:13.670 - Oh, sorry. Of course. 84 00:04:13.670 --> 00:04:14.850 Yeah. 85 00:04:14.850 --> 00:04:18.230 I will try and speak louder, closer to the microphone, 86 00:04:18.230 --> 00:04:23.173 and a little bit more slowly and keep an eye on that. 87 00:04:24.430 --> 00:04:25.523 I'm sorry about that. 88 00:04:28.590 --> 00:04:32.458 So what you may have missed was sort 89 00:04:32.458 --> 00:04:35.430 of a high-level overview of what we're about to talk 90 00:04:35.430 --> 00:04:38.160 about today, which is about the statewide data 91 00:04:38.160 --> 00:04:42.630 indicators and the measurement and sampling approach, 92 00:04:42.630 --> 00:04:44.480 both in associated position 93 00:04:44.480 --> 00:04:46.997 with the CFSR review itself; next week, 94 00:04:46.997 --> 00:04:49.280 the focus is on the PIP. 95 00:04:49.280 --> 00:04:51.430 And one thing that I did note and want to reiterate 96 00:04:51.430 --> 00:04:54.490 is the substance of what we're presenting today, 97 00:04:54.490 --> 00:04:56.920 we wanted to present it in a way 98 00:04:56.920 --> 00:04:59.920 that accommodated your questions and elaboration 99 00:04:59.920 --> 00:05:02.140 of information, all of which should be, 100 00:05:02.140 --> 00:05:06.730 is in Technical Bulletin #13, which is available 101 00:05:06.730 --> 00:05:10.600 to you through the CFSR Information Portal, 102 00:05:10.600 --> 00:05:13.509 and we can make sure that you have that information, 103 00:05:13.509 --> 00:05:16.760 it's I believe associated with the resources 104 00:05:16.760 --> 00:05:19.403 that I mentioned on the slides. 105 00:05:20.390 --> 00:05:21.223 Okay. 106 00:05:21.223 --> 00:05:23.520 So I wanted to just give a quick overview 107 00:05:23.520 --> 00:05:26.320 of the indicators themselves, 108 00:05:26.320 --> 00:05:30.300 remind you that they were developed to, well, 109 00:05:30.300 --> 00:05:32.820 the statewide data indicators have been part 110 00:05:32.820 --> 00:05:35.800 of the CFSR process from its inception. 111 00:05:35.800 --> 00:05:40.650 They were revised before Round 3 and ultimately were 112 00:05:40.650 --> 00:05:44.140 not used in the determinations of substantial conformity, 113 00:05:44.140 --> 00:05:46.870 although they continued to be generated and used 114 00:05:46.870 --> 00:05:49.848 over the course of that round. 115 00:05:49.848 --> 00:05:54.430 And informed both discussions about performance 116 00:05:54.430 --> 00:05:55.380 as part of the review, 117 00:05:55.380 --> 00:05:59.970 and then during the PIP implementation and review process. 118 00:05:59.970 --> 00:06:04.970 We are, we, I don't know what you reckon. 119 00:06:05.300 --> 00:06:06.450 We are returning 120 00:06:06.450 --> 00:06:11.428 them to their role in CFSR Round 4, as we have been saying, 121 00:06:11.428 --> 00:06:15.420 and they will be one key way that we are able 122 00:06:15.420 --> 00:06:19.921 to indicate the extent to which states are performing, 123 00:06:19.921 --> 00:06:23.520 what their performance is on the desired outcomes 124 00:06:23.520 --> 00:06:25.596 captured within those indicators. 125 00:06:25.596 --> 00:06:28.550 There are seven of them. We'll get to them in a minute. 126 00:06:28.550 --> 00:06:32.060 Five on permanency and two that look to safety 127 00:06:32.060 --> 00:06:34.150 for children who have come into contact 128 00:06:34.150 --> 00:06:35.650 with the child welfare system. 129 00:06:36.790 --> 00:06:38.940 The performance in the statewide indicators 130 00:06:39.790 --> 00:06:42.730 is generated in a manner that lets you do two things. 131 00:06:42.730 --> 00:06:45.820 First it tells you the observed performance. 132 00:06:45.820 --> 00:06:48.040 What have we observed in recent years 133 00:06:48.040 --> 00:06:53.040 for your state's performance on those seven indicators? 134 00:06:53.128 --> 00:06:57.250 We also supplement them with information that allows 135 00:06:57.250 --> 00:06:59.040 you to see how your performance, 136 00:06:59.040 --> 00:07:03.560 a state's performance is relative to national performance. 137 00:07:03.560 --> 00:07:07.576 And it is that second approach that lets us make 138 00:07:07.576 --> 00:07:11.997 a determination about a state's performance relative 139 00:07:11.997 --> 00:07:15.620 to observed national performance. 140 00:07:15.620 --> 00:07:18.280 The statewide data indicators are calculated using 141 00:07:18.280 --> 00:07:21.630 AFCARS data and NCANDS data, 142 00:07:21.630 --> 00:07:24.393 and one of those metrics uses both of them. 143 00:07:25.980 --> 00:07:27.540 So let me just go over them quickly, 144 00:07:27.540 --> 00:07:32.288 and then I can turn this over to Roger. 145 00:07:32.288 --> 00:07:36.660 The first two measures 146 00:07:36.660 --> 00:07:39.560 are associated with the Safety outcome, 147 00:07:39.560 --> 00:07:42.770 which as specified in the regulation is an outcome 148 00:07:42.770 --> 00:07:45.010 where we are interested in understanding the extent 149 00:07:45.010 --> 00:07:48.290 to which children are, first and foremost, 150 00:07:48.290 --> 00:07:50.370 protected from abuse and neglect. 151 00:07:50.370 --> 00:07:52.960 This first measure, maltreatment in care, 152 00:07:52.960 --> 00:07:57.250 is a rate and it measures the rate of abuse or neglect 153 00:07:57.250 --> 00:08:01.740 or the rate of substantiated allegation, 154 00:08:01.740 --> 00:08:03.880 a child who is in the care, in custody 155 00:08:03.880 --> 00:08:07.530 of a state experienced per number of days in foster 156 00:08:07.530 --> 00:08:11.810 care that all children in care experienced 157 00:08:11.810 --> 00:08:13.780 during the year under review. 158 00:08:13.780 --> 00:08:18.070 So I will note that I possibly did not say 159 00:08:18.070 --> 00:08:19.740 that as elegantly as possible, 160 00:08:19.740 --> 00:08:24.740 but we do have the definitions and the denominators, 161 00:08:24.891 --> 00:08:26.820 I think the numerators and denominators for each 162 00:08:26.820 --> 00:08:30.363 of these measures are appended to Technical Bulletin #13. 163 00:08:31.400 --> 00:08:32.490 The second safety measure 164 00:08:32.490 --> 00:08:34.680 is the recurrence of maltreatment. 165 00:08:34.680 --> 00:08:36.800 This is a percent, not a rate, 166 00:08:36.800 --> 00:08:39.470 and it looks at the percent of children who were 167 00:08:39.470 --> 00:08:42.130 the subject of the substantiated or indicated report 168 00:08:42.130 --> 00:08:45.590 of maltreatment in a given 12-month period, 169 00:08:45.590 --> 00:08:49.040 who then experienced a subsequent maltreatment 170 00:08:50.066 --> 00:08:53.610 within the subsequent 12 months following 171 00:08:53.610 --> 00:08:55.520 the initial victimization. 172 00:08:55.520 --> 00:08:59.310 So children who were victimized during a 1-year period, 173 00:08:59.310 --> 00:09:02.008 the proportion of them who were re-victimized 174 00:09:02.008 --> 00:09:05.963 within 12 months of their original victimization. 175 00:09:07.270 --> 00:09:11.550 The next five measures relate to permanency, 176 00:09:11.550 --> 00:09:14.880 and are intended to evaluate the extent 177 00:09:14.880 --> 00:09:17.900 to which children have permanency and stability 178 00:09:17.900 --> 00:09:20.430 in their living situations. 179 00:09:20.430 --> 00:09:22.768 The first three of those measures look 180 00:09:22.768 --> 00:09:25.798 at the experience of children, 181 00:09:25.798 --> 00:09:28.520 look at the exit experiences of children who are 182 00:09:28.520 --> 00:09:32.230 in the care of the state child welfare agency. 183 00:09:32.230 --> 00:09:34.540 The first is an entry cohort measure, 184 00:09:34.540 --> 00:09:37.990 and it looks among children entering care in a 12- 185 00:09:37.990 --> 00:09:39.960 month period, 186 00:09:39.960 --> 00:09:42.939 it looks to see the percent of those children 187 00:09:42.939 --> 00:09:47.939 who exited to a permanent exit of some kind, reunification, 188 00:09:48.600 --> 00:09:52.500 adoption, guardianship, or living with a relative, 189 00:09:52.500 --> 00:09:55.860 within 12 months of their entry. 190 00:09:55.860 --> 00:09:57.670 So among the kids entering care, 191 00:09:57.670 --> 00:10:00.290 the proportion that exited to permanency within 12 192 00:10:00.290 --> 00:10:01.583 months of their entry. 193 00:10:03.240 --> 00:10:05.980 The second two permanency measures consider a group 194 00:10:05.980 --> 00:10:08.430 of children who were in care at the first day 195 00:10:08.430 --> 00:10:10.460 of a 12-month period, 196 00:10:10.460 --> 00:10:13.050 measure four looks at among children who were in care 197 00:10:13.050 --> 00:10:16.540 on the first day of a 12- month period and had been 198 00:10:16.540 --> 00:10:21.540 in care between 12 and 23 months already, as of that date, 199 00:10:21.550 --> 00:10:24.380 the proportion of those children who exited 200 00:10:24.380 --> 00:10:27.325 to permanency of some kind in the next, 201 00:10:27.325 --> 00:10:30.960 in the subsequent or the next 12-month period. 202 00:10:30.960 --> 00:10:34.003 So if you think about it as the start of a fiscal year, 203 00:10:34.950 --> 00:10:35.860 which it could be, 204 00:10:35.860 --> 00:10:38.410 among children in care on the first day of the fiscal year 205 00:10:38.410 --> 00:10:42.150 who had already been in care between 12 and 23 months 206 00:10:42.150 --> 00:10:45.880 of that day, the proportion of that group who exited 207 00:10:45.880 --> 00:10:49.073 to permanency by the last day of the fiscal year. 208 00:10:50.290 --> 00:10:53.620 The fifth measure, which is the third permanency measure, 209 00:10:53.620 --> 00:10:56.130 looks at a similar group of children, 210 00:10:56.130 --> 00:10:59.200 but what distinguishes them is that on the first 211 00:10:59.200 --> 00:11:01.250 day of the 12-month period, 212 00:11:01.250 --> 00:11:04.390 they had already been in care for 24 months 213 00:11:04.390 --> 00:11:06.630 or 2 years, at least. 214 00:11:06.630 --> 00:11:09.760 And it then considers the proportion of them who exit 215 00:11:09.760 --> 00:11:11.853 in the subsequent 12-month period. 216 00:11:13.190 --> 00:11:18.190 The sixth measure is, measures reentry, 217 00:11:18.240 --> 00:11:20.425 and it looks at a group of children who discharge 218 00:11:20.425 --> 00:11:22.853 to permanency or exit to permanency, 219 00:11:23.870 --> 00:11:26.770 not including adoption, but any other permanent exit 220 00:11:26.770 --> 00:11:30.214 that happened in a 12-month period. 221 00:11:30.214 --> 00:11:31.651 So it's an exit group. 222 00:11:31.651 --> 00:11:34.580 And looks to see the proportion of those kids, 223 00:11:34.580 --> 00:11:37.206 of the percent of those who re-enter care within 12 224 00:11:37.206 --> 00:11:39.173 months of their exit. 225 00:11:40.770 --> 00:11:43.180 And then finally, the seventh and final measure. 226 00:11:43.180 --> 00:11:44.420 And the last permanency measure 227 00:11:44.420 --> 00:11:46.410 is a measure, placement stability. 228 00:11:46.410 --> 00:11:48.530 And it is also a rate. 229 00:11:48.530 --> 00:11:51.300 It looks, again, at a group of kids who enter care 230 00:11:51.300 --> 00:11:53.490 in a 12-month period, 231 00:11:53.490 --> 00:11:56.440 and it considers the number of placement moves 232 00:11:56.440 --> 00:12:01.129 that group had for the number of days 233 00:12:01.129 --> 00:12:03.160 they all experienced during that year. 234 00:12:03.160 --> 00:12:04.760 So I can say that better. 235 00:12:04.760 --> 00:12:07.420 So among the group of children who enter care 236 00:12:07.420 --> 00:12:09.060 in a 12-month period, 237 00:12:09.060 --> 00:12:11.970 this rate is generated by looking at all of the days 238 00:12:11.970 --> 00:12:13.800 that group of children used, 239 00:12:13.800 --> 00:12:15.940 so a child who entered care in the beginning 240 00:12:15.940 --> 00:12:17.920 of the year and still in the care at the end 241 00:12:17.920 --> 00:12:21.540 of the year would have a lot of days, say over 300 days, 242 00:12:21.540 --> 00:12:23.920 a child who entered care at the end of the year, 243 00:12:23.920 --> 00:12:27.040 maybe the last, in the last 2 weeks of the year 244 00:12:27.040 --> 00:12:30.490 would have 2 weeks in the denominator. 245 00:12:30.490 --> 00:12:35.320 All of those days are the denominator, and the number of moves, 246 00:12:35.320 --> 00:12:38.390 each placement moves each of those children experience 247 00:12:38.390 --> 00:12:39.770 in the year is the numerator, 248 00:12:39.770 --> 00:12:41.960 and that gives us the rate, 249 00:12:41.960 --> 00:12:45.283 which we multiply by 1000 to make more interpretable. 250 00:12:46.140 --> 00:12:48.550 But that's not really a high-level view. 251 00:12:48.550 --> 00:12:51.640 That's a little bit more of an "in the details" view, 252 00:12:51.640 --> 00:12:56.327 which is my signal to turn it over to Roger, who will 253 00:12:56.327 --> 00:13:00.410 go into more detail about these metrics and how they'll 254 00:13:00.410 --> 00:13:05.390 be used to support Round 4 of the CFSR. 255 00:13:05.390 --> 00:13:06.223 Roger? 256 00:13:08.650 --> 00:13:09.727 - Thanks, Jennifer. 257 00:13:12.800 --> 00:13:13.633 Okay. 258 00:13:13.633 --> 00:13:17.143 So Jennifer just explained the seven indicators 259 00:13:17.143 --> 00:13:20.370 that we'll be using for Round 4. 260 00:13:20.370 --> 00:13:21.960 I just want to go into some of the details 261 00:13:21.960 --> 00:13:24.423 on how those indicators are used. 262 00:13:25.590 --> 00:13:29.530 First, and I think Jennifer noted in Round 4, 263 00:13:29.530 --> 00:13:32.800 the re-entry measure that was just described 264 00:13:32.800 --> 00:13:34.760 is a new measure. 265 00:13:34.760 --> 00:13:39.320 So in Round 3, re-entry was a companion measure 266 00:13:39.320 --> 00:13:43.510 to permanency in 12 months for entry, and as such 267 00:13:43.510 --> 00:13:44.920 it measured re-entry only 268 00:13:44.920 --> 00:13:47.590 for those children who discharged within 12 269 00:13:47.590 --> 00:13:48.673 months of entry. 270 00:13:50.360 --> 00:13:54.071 And so it would be the denominator for the measure, 271 00:13:54.071 --> 00:13:57.616 that is children who were at risk of re-entry, 272 00:13:57.616 --> 00:14:01.130 were only those children who had exited care 273 00:14:01.130 --> 00:14:03.130 within 12 months of their entry. 274 00:14:03.130 --> 00:14:06.580 Further, that was limited to children who 275 00:14:06.580 --> 00:14:09.663 were in care for at least 8 days. 276 00:14:10.850 --> 00:14:13.995 So the new measure is based on an exit cohort 277 00:14:13.995 --> 00:14:18.995 and it will be all children who are at risk of re-entry, 278 00:14:19.290 --> 00:14:22.200 or all children who exit in a year. 279 00:14:22.200 --> 00:14:24.530 And then we will follow them for another 12 months 280 00:14:24.530 --> 00:14:26.130 to see if they re-enter care. 281 00:14:26.130 --> 00:14:29.960 And so this will be regardless of how long their stay 282 00:14:29.960 --> 00:14:33.166 in foster care was, whether it was 2 days or 2 years, 283 00:14:33.166 --> 00:14:36.200 they will be included to check for whether or not 284 00:14:36.200 --> 00:14:37.653 the child re-entered care. 285 00:14:38.670 --> 00:14:42.173 Okay. Another update is to the data indicators - 286 00:14:42.173 --> 00:14:47.173 we will continue to use the trial home visit as in Round 3. 287 00:14:47.440 --> 00:14:48.970 However, we are shortening that. 288 00:14:48.970 --> 00:14:53.970 So previously we looked at 12 months beyond the 2 289 00:14:54.370 --> 00:14:57.550 years required to observe entries and exits. 290 00:14:57.550 --> 00:14:59.900 And we have shortened that to 6 months. 291 00:14:59.900 --> 00:15:04.230 So it is now a 6-month trial home visit view to make 292 00:15:04.230 --> 00:15:05.990 the trial home visit adjustment. 293 00:15:05.990 --> 00:15:09.257 And that will allow for us to report on the permanency 294 00:15:09.257 --> 00:15:11.820 in 12 months re-entries measure 295 00:15:11.820 --> 00:15:14.323 6 months sooner than we were able to before. 296 00:15:16.310 --> 00:15:20.890 There are also some changes to the data quality checks; 297 00:15:20.890 --> 00:15:22.930 these are pretty minor, 298 00:15:22.930 --> 00:15:27.930 but they result in essentially fewer or limited 299 00:15:28.160 --> 00:15:29.250 data quality checks. 300 00:15:29.250 --> 00:15:32.770 So the changes are that we will no longer apply 301 00:15:32.770 --> 00:15:36.133 data quality checks to people 21 years or over. 302 00:15:38.020 --> 00:15:41.860 And also we are eliminating data quality checks 303 00:15:41.860 --> 00:15:45.300 within a submission period that have, 304 00:15:45.300 --> 00:15:48.560 for records that have activity that occurred outside 305 00:15:48.560 --> 00:15:50.310 of that submission period, or sorry, 306 00:15:50.310 --> 00:15:53.083 outside of the period for the indicator. 307 00:15:54.790 --> 00:15:59.105 All of these changes that I'm describing are detailed 308 00:15:59.105 --> 00:16:01.917 in Technical Bulletin #13, if you want 309 00:16:01.917 --> 00:16:04.020 additional information or want to review, 310 00:16:04.020 --> 00:16:06.320 those are available in Technical Bulletin #13. 311 00:16:07.870 --> 00:16:12.210 And we will update the syntax to calculate 312 00:16:12.210 --> 00:16:13.550 the performance on the indicator. 313 00:16:13.550 --> 00:16:17.430 So we do for Round 3, we released the syntax used 314 00:16:17.430 --> 00:16:19.960 to calculate the performance. 315 00:16:19.960 --> 00:16:24.960 We are updating that and we'll release the revised syntax 316 00:16:25.460 --> 00:16:28.100 in spring of 2022. 317 00:16:28.100 --> 00:16:30.450 And those will be made available to all states. 318 00:16:34.240 --> 00:16:36.980 So next, I'll explain how performance 319 00:16:36.980 --> 00:16:39.650 on the statewide data indicators is reported and used 320 00:16:39.650 --> 00:16:42.540 in the determination of substantial conformity 321 00:16:42.540 --> 00:16:46.310 for Safety Outcome 1 and Permanency Outcome 1, 322 00:16:46.310 --> 00:16:49.040 and describe some of the terminology we use 323 00:16:49.040 --> 00:16:51.080 to describe this performance. 324 00:16:51.080 --> 00:16:56.080 So [unintell.] we use to describe observed performance 325 00:16:57.120 --> 00:16:59.823 on an indicator for the entire nation. 326 00:17:00.660 --> 00:17:03.650 So for example, for the indicator 327 00:17:03.650 --> 00:17:06.110 for "permanency in 12 months for children entering 328 00:17:06.110 --> 00:17:10.150 foster care," the national performance is based 329 00:17:10.150 --> 00:17:12.800 on all children who entered care across the nation 330 00:17:12.800 --> 00:17:15.334 in 1 year, and then how many of them exited 331 00:17:15.334 --> 00:17:17.770 within 12 months of their entry. 332 00:17:17.770 --> 00:17:21.853 So it's the sum of performance across all states, 333 00:17:24.200 --> 00:17:26.810 and we will calculate national performance 334 00:17:26.810 --> 00:17:30.660 for each indicator prior to the start of Round 4. 335 00:17:30.660 --> 00:17:33.370 And that will, those values will remain fixed 336 00:17:33.370 --> 00:17:34.310 throughout Round 4. 337 00:17:34.310 --> 00:17:37.740 So this is the same as what we did in Round 3 338 00:17:38.850 --> 00:17:43.730 in that the indicators, 339 00:17:43.730 --> 00:17:45.090 the performance, 340 00:17:45.090 --> 00:17:47.630 national performance will not change throughout the round. 341 00:17:47.630 --> 00:17:50.130 So with each new data submission 342 00:17:50.130 --> 00:17:52.770 through AFCARS and NCANDS, 343 00:17:52.770 --> 00:17:54.050 state performance changes 344 00:17:54.050 --> 00:17:56.200 every 6 months as we update the data, 345 00:17:56.200 --> 00:17:58.910 but the national performance values will remain 346 00:17:58.910 --> 00:18:01.143 fixed throughout the round. 347 00:18:03.118 --> 00:18:06.270 A few questions I missed, somebody asked previously, 348 00:18:06.270 --> 00:18:09.120 will the syntax be available in SAS? 349 00:18:09.120 --> 00:18:10.439 It will not be, 350 00:18:10.439 --> 00:18:14.453 we have it in SPSS and we are releasing that. 351 00:18:15.460 --> 00:18:18.340 And I don't know a lot of the details, 352 00:18:18.340 --> 00:18:22.280 but I believe the Center for States is available 353 00:18:22.280 --> 00:18:26.010 to help people as they try to translate that syntax 354 00:18:26.010 --> 00:18:29.923 from SPSS to other programs. 355 00:18:32.710 --> 00:18:35.913 so also national performance, 356 00:18:36.790 --> 00:18:40.550 those values will be released in an update 357 00:18:40.550 --> 00:18:41.760 to Technical Bulletin #13. 358 00:18:41.760 --> 00:18:44.240 So we will provide what all the national 359 00:18:44.240 --> 00:18:47.130 performance values are in the update 360 00:18:47.130 --> 00:18:48.513 to Technical Bulletin #13. 361 00:18:49.960 --> 00:18:52.483 So then also another performance measure 362 00:18:52.483 --> 00:18:55.830 is what we refer to as Risk- Standardized Performance. 363 00:18:55.830 --> 00:18:58.170 It is a [unintell.] measure of a state's performance 364 00:18:58.170 --> 00:19:01.250 after risk adjustment, where that risk adjustment 365 00:19:01.250 --> 00:19:04.270 is a [unintell.] process that accounts for some 366 00:19:04.270 --> 00:19:07.590 of the factors outside of the state's control, 367 00:19:07.590 --> 00:19:10.363 such as the age of children in foster care. 368 00:19:11.480 --> 00:19:14.626 So we calculate Risk- Standardized Performance 369 00:19:14.626 --> 00:19:19.480 for each indicator, and every state has a unique 370 00:19:19.480 --> 00:19:22.027 Risk-Standardized Performance value for their indicator. 371 00:19:22.027 --> 00:19:24.450 And it is Risk-Standardized Performance 372 00:19:24.450 --> 00:19:27.140 that identifies whether a state's performance 373 00:19:27.140 --> 00:19:30.410 on a data indicator is statistically worse, 374 00:19:30.410 --> 00:19:33.323 no different or better than national performance. 375 00:19:37.780 --> 00:19:38.613 Okay. 376 00:19:38.613 --> 00:19:43.260 And all of these performance values are made available 377 00:19:44.380 --> 00:19:48.890 in the data profiles that we release every 6 months. 378 00:19:48.890 --> 00:19:52.060 And those data profiles include the Risk- 379 00:19:52.060 --> 00:19:54.670 Standardized Performance values, the state 380 00:19:54.670 --> 00:19:57.690 Observed Performance values, they also show 381 00:19:57.690 --> 00:20:01.290 the national performance value for each data indicator 382 00:20:01.290 --> 00:20:02.367 and have the results 383 00:20:02.367 --> 00:20:04.813 of the various data quality checks. 384 00:20:08.199 --> 00:20:11.652 And so the profiles are released every 6 months 385 00:20:11.652 --> 00:20:16.652 in roughly February and August, and those releases reflect 386 00:20:17.680 --> 00:20:20.950 the most recent updates or most recent data submitted 387 00:20:20.950 --> 00:20:22.193 to AFCARS and NCANDS. 388 00:20:24.890 --> 00:20:28.600 Along with the data profiles, 389 00:20:28.600 --> 00:20:31.050 we also provide what we refer 390 00:20:31.050 --> 00:20:33.320 to as supplemental context data. 391 00:20:33.320 --> 00:20:36.370 This comes in the form of an Excel file, 392 00:20:36.370 --> 00:20:40.200 which has within it some basic characteristics 393 00:20:40.200 --> 00:20:43.670 about the states and national foster care populations. 394 00:20:43.670 --> 00:20:46.883 So we have in there the state entry rates, 395 00:20:47.910 --> 00:20:51.750 some other performance measures that we show, 396 00:20:51.750 --> 00:20:54.653 that show discharges to permanency by exit type. 397 00:20:56.000 --> 00:21:01.000 The entry rates, for example, are also disaggregated by age, 398 00:21:01.670 --> 00:21:05.863 race, ethnicity, and county or other locality. 399 00:21:08.670 --> 00:21:11.960 And then we also include, for each state, 400 00:21:11.960 --> 00:21:15.680 we have tabs in the supplemental context data 401 00:21:15.680 --> 00:21:18.873 that disaggregate each indicator by age, race, 402 00:21:18.873 --> 00:21:21.370 ethnicity, and county. 403 00:21:21.370 --> 00:21:25.670 So a state's supplemental context file has the entry 404 00:21:25.670 --> 00:21:27.550 rates and some other analyses, 405 00:21:27.550 --> 00:21:31.780 and then has each indicator disaggregated by age, race, 406 00:21:31.780 --> 00:21:33.354 ethnicity, and county. 407 00:21:33.354 --> 00:21:37.790 And we also release a national workbook that has all 408 00:21:37.790 --> 00:21:42.503 of the same information, but for the nation as a whole. 409 00:21:45.170 --> 00:21:48.400 And these supplemental context data files are, 410 00:21:48.400 --> 00:21:50.410 provide information to help states identify 411 00:21:50.410 --> 00:21:53.130 patterns across subgroups, so that's, 412 00:21:53.130 --> 00:21:56.064 their sort of main purpose is to help provide 413 00:21:56.064 --> 00:21:58.000 additional information on the, 414 00:21:58.000 --> 00:22:00.000 on what's contained in the data profile. 415 00:22:07.812 --> 00:22:11.729 So next, in determining substantial conformity, 416 00:22:12.910 --> 00:22:17.910 we will use the Round 3 plan in Round 4. 417 00:22:18.510 --> 00:22:21.510 So performance on the data indicators 418 00:22:21.510 --> 00:22:23.620 along with case review performance will be used 419 00:22:23.620 --> 00:22:26.115 to determine substantial conformity 420 00:22:26.115 --> 00:22:28.973 for Safety Outcome 1 and Permanency Outcome 1. 421 00:22:30.980 --> 00:22:33.618 And the way that this will work is that if a state's 422 00:22:33.618 --> 00:22:37.520 Risk-Standardized Performance on a data indicator 423 00:22:37.520 --> 00:22:39.560 is typically worse than national performance 424 00:22:39.560 --> 00:22:41.100 as indicated in the data profile, 425 00:22:41.100 --> 00:22:43.820 then the state is not in substantial conformity 426 00:22:43.820 --> 00:22:44.940 with the associated outcome. 427 00:22:44.940 --> 00:22:49.700 And so the data profile will indicate the RSP values, 428 00:22:52.050 --> 00:22:53.033 but those, if you are familiar with the data profiles, 429 00:22:53.033 --> 00:22:55.252 they're also color-coded. 430 00:22:55.252 --> 00:22:58.950 So if it is orange or red, 431 00:22:58.950 --> 00:23:00.650 depending on how you see that color, 432 00:23:02.010 --> 00:23:07.010 that indicates that the state is statistically worse 433 00:23:07.050 --> 00:23:09.250 than national performance on that indicator. 434 00:23:10.250 --> 00:23:13.720 Also for Round 4, we will be releasing, there will be 435 00:23:13.720 --> 00:23:15.857 a slight update to the data profiles 436 00:23:15.857 --> 00:23:19.990 that will include a revised cover page that should 437 00:23:19.990 --> 00:23:24.375 make RSP a little easier to understand and see 438 00:23:24.375 --> 00:23:26.653 the performance results there. 439 00:23:28.386 --> 00:23:29.540 And so in the determination of substantial conformity, 440 00:23:29.540 --> 00:23:32.330 it is based on the most recent performance 441 00:23:32.330 --> 00:23:34.610 in the data profile transmitted 442 00:23:34.610 --> 00:23:36.060 for the statewide assessment. 443 00:23:36.953 --> 00:23:38.080 And then in the Final Report, 444 00:23:38.080 --> 00:23:39.971 we will include comments or recommendations 445 00:23:39.971 --> 00:23:43.960 about performance changes reflected in the data profile 446 00:23:43.960 --> 00:23:46.123 to states after the statewide assessment. 447 00:23:47.370 --> 00:23:50.310 Also, any data quality issues, 448 00:23:50.310 --> 00:23:52.570 any DQ issues that prevent performance 449 00:23:52.570 --> 00:23:55.000 from being calculated on an indicator - 450 00:23:55.000 --> 00:24:00.000 so if a data quality check is failed, 451 00:24:01.230 --> 00:24:03.970 that can often generally prevent at least one 452 00:24:03.970 --> 00:24:07.280 indicator from being calculated for at least one 453 00:24:07.280 --> 00:24:11.820 time period. That can also have a cascading effect, 454 00:24:11.820 --> 00:24:15.630 such that a single data quality issue 455 00:24:15.630 --> 00:24:17.660 in a single AFCARS submission, 456 00:24:17.660 --> 00:24:21.640 for example, can cascade and impact multiple 457 00:24:21.640 --> 00:24:24.173 indicators across multiple time periods. 458 00:24:25.200 --> 00:24:26.940 And so if there is a data quality issue 459 00:24:26.940 --> 00:24:29.490 that prevents performance from being calculated, 460 00:24:29.490 --> 00:24:32.653 that will result in a determination of nonconformity. 461 00:24:35.500 --> 00:24:40.500 And then independent of CFSR Round 4, but still 462 00:24:40.978 --> 00:24:43.190 coinciding at the same time, 463 00:24:43.190 --> 00:24:45.620 as overlapping really in time with CFSR Round 4, 464 00:24:45.620 --> 00:24:50.350 is the new AFCARS 2020 reporting system. 465 00:24:50.350 --> 00:24:54.500 The first submissions for AFCARS 2020 are due May 466 00:24:54.500 --> 00:24:58.564 15th, 2023. That is independent 467 00:24:58.564 --> 00:25:01.390 of CFSR, but that does coincide. 468 00:25:01.390 --> 00:25:05.040 And so we will calculate performance on the indicators, 469 00:25:05.040 --> 00:25:07.580 the Children's Bureau will map data elements, 470 00:25:07.580 --> 00:25:11.703 the sort of new or changed data elements in AFCARS 471 00:25:11.703 --> 00:25:13.970 2020 will be mapped to the data elements 472 00:25:13.970 --> 00:25:16.253 for AFCARS 1993. 473 00:25:21.050 --> 00:25:26.050 And lastly, then, some of the additional resources 474 00:25:26.150 --> 00:25:29.450 that are available, as I mentioned, 475 00:25:29.450 --> 00:25:30.540 we release the data profile, 476 00:25:30.540 --> 00:25:31.810 the context data that will be released 477 00:25:31.810 --> 00:25:35.613 on the same schedule as it was in CFSR Round 3, 478 00:25:37.810 --> 00:25:41.053 Technical Bulletin #13 is available, 479 00:25:42.360 --> 00:25:44.170 clickable here in this presentation, 480 00:25:44.170 --> 00:25:45.370 which can be downloaded. 481 00:25:46.740 --> 00:25:48.657 And that has a lot of the details 482 00:25:48.657 --> 00:25:51.453 that we have gone over regarding the indicators. 483 00:25:52.710 --> 00:25:54.870 It will have the updated performance values 484 00:25:54.870 --> 00:25:56.760 for national performance. 485 00:25:56.760 --> 00:26:01.410 There's also the Capacity Building Center's toolkit, 486 00:26:01.410 --> 00:26:04.456 which has a quick reference for the data profiles 487 00:26:04.456 --> 00:26:08.363 and fact sheets, some additional information there, 488 00:26:09.500 --> 00:26:13.030 and also the Children's Bureau CFSR Information Portal, 489 00:26:13.030 --> 00:26:15.559 which has some additional information. 490 00:26:15.559 --> 00:26:19.700 For the most part, none of these resources are really new, 491 00:26:19.700 --> 00:26:23.350 except for Technical Bulletin #13 and information 492 00:26:23.350 --> 00:26:27.380 that is specific to Round 4, but otherwise the fact 493 00:26:27.380 --> 00:26:28.460 sheets on the indicators, 494 00:26:28.460 --> 00:26:31.333 a lot of that information is unchanged. 495 00:26:33.970 --> 00:26:35.500 Okay. So we can take time for questions. 496 00:26:35.500 --> 00:26:37.990 I realized that several questions came in, 497 00:26:37.990 --> 00:26:40.090 sort of about specific slides. 498 00:26:40.090 --> 00:26:41.470 It's, to be honest, 499 00:26:41.470 --> 00:26:43.550 a little difficult to stop and follow some of those. 500 00:26:43.550 --> 00:26:45.540 So I'll try to go back through and answer questions. 501 00:26:45.540 --> 00:26:47.985 I answered the question about whether the syntax 502 00:26:47.985 --> 00:26:49.843 would be available in SAS. 503 00:26:51.053 --> 00:26:53.310 Another question is what years 504 00:26:53.310 --> 00:26:57.360 are used to calculate the observed national performance? 505 00:26:57.360 --> 00:27:00.240 It's a good question, but a complicated one, honestly, 506 00:27:00.240 --> 00:27:02.470 I can't remember off the top of my head which years 507 00:27:02.470 --> 00:27:03.720 are used for each of them. 508 00:27:06.130 --> 00:27:08.420 Each of the indicators sort of has its own set 509 00:27:08.420 --> 00:27:10.140 of years that are used. 510 00:27:10.140 --> 00:27:14.670 And that depends on how many years are required 511 00:27:14.670 --> 00:27:16.230 to calculate performance. 512 00:27:16.230 --> 00:27:19.280 So, for example, to calculate placement stability, 513 00:27:19.280 --> 00:27:21.960 we only need two AFCARS periods. 514 00:27:21.960 --> 00:27:23.820 And so for national performance, 515 00:27:23.820 --> 00:27:26.680 that will based on the most recent two periods 516 00:27:26.680 --> 00:27:30.320 that are available, but for permanency for entry, for example, 517 00:27:30.320 --> 00:27:31.290 we need five AFCARS periods. 518 00:27:31.290 --> 00:27:34.690 So that has to go back a year and a half earlier 519 00:27:34.690 --> 00:27:39.057 than what we would use for substantial conformity. 520 00:27:42.720 --> 00:27:45.460 Another question is, will a state-by-state comparison 521 00:27:45.460 --> 00:27:49.283 be provided to know how we fare across measures nationally? 522 00:27:51.100 --> 00:27:53.023 To my knowledge, we are not planning to release any 523 00:27:53.023 --> 00:27:56.060 sort of state-by-state comparison. 524 00:27:56.060 --> 00:27:59.660 In general, we sort of discourage comparing states 525 00:27:59.660 --> 00:28:00.723 to other states. 526 00:28:03.460 --> 00:28:05.850 We have Risk-Standardized Performance, 527 00:28:05.850 --> 00:28:07.440 which is designed to compare a state 528 00:28:07.440 --> 00:28:08.840 to the national performance. 529 00:28:10.900 --> 00:28:13.410 And we do release the context data workbook 530 00:28:13.410 --> 00:28:15.163 for national performance. 531 00:28:16.100 --> 00:28:20.920 There is a workbook that has all of the values 532 00:28:20.920 --> 00:28:23.303 for each state for standard performance. 533 00:28:24.350 --> 00:28:27.383 And it has standard performance and national performance in it. 534 00:28:28.460 --> 00:28:33.370 That said, comparing states' observed performance 535 00:28:33.370 --> 00:28:36.600 to each other... [unintell.] 536 00:28:38.630 --> 00:28:42.200 And so that resource is available, 537 00:28:42.200 --> 00:28:46.040 but it doesn't provide the sort of direct comparison. 538 00:28:46.040 --> 00:28:48.300 - [Elizabeth] Roger, we do have the state-by-state workbook 539 00:28:48.300 --> 00:28:50.810 that's posted on the CFSR portal that does 540 00:28:50.810 --> 00:28:52.893 have every state. 541 00:28:53.860 --> 00:28:54.693 - [Roger] Okay. Right. 542 00:28:54.693 --> 00:28:56.700 So it, yes, the workbook, portal, I'm sorry. 543 00:28:56.700 --> 00:28:58.730 On the portal, the workbook has each state's 544 00:28:58.730 --> 00:29:02.003 Risk-Standardized Performance and Observed Performance, I believe. 545 00:29:03.970 --> 00:29:04.950 - [Elizabeth] Yeah, that's right. 546 00:29:04.950 --> 00:29:07.350 And we publish that once a year, 547 00:29:07.350 --> 00:29:09.113 and with the August profile. 548 00:29:10.020 --> 00:29:12.100 - [Roger] But I think we generally discourage states 549 00:29:12.100 --> 00:29:14.750 comparing themselves to other states. 550 00:29:14.750 --> 00:29:15.973 - [Elizabeth] That's right. 551 00:29:20.670 --> 00:29:22.210 - Another question, can you say more 552 00:29:22.210 --> 00:29:24.050 about Risk-Standardized Performance? 553 00:29:24.050 --> 00:29:26.893 What factors are included in the calculation? 554 00:29:28.430 --> 00:29:32.730 So in the resources that are available 555 00:29:32.730 --> 00:29:34.820 through the Center for the - 556 00:29:34.820 --> 00:29:36.957 the Capacity Building Center for States, 557 00:29:36.957 --> 00:29:41.020 there is an elaborate description 558 00:29:41.020 --> 00:29:45.560 of Risk-Standardized Performance, how it is calculated, 559 00:29:45.560 --> 00:29:47.213 and how it is used. 560 00:29:48.950 --> 00:29:52.050 The factors that are included in the calculation do 561 00:29:52.050 --> 00:29:53.993 vary by indicator. 562 00:29:54.970 --> 00:29:59.240 So for example, permanency for entries, I believe, 563 00:29:59.240 --> 00:30:01.893 uses entry rate and age at entry. 564 00:30:03.229 --> 00:30:07.220 And some other indicators use different values, or sorry, 565 00:30:07.220 --> 00:30:12.220 different factors for the calculation. 566 00:30:12.800 --> 00:30:14.373 In general, Risk-Standardized Performance 567 00:30:14.373 --> 00:30:18.500 is a ratio of the observed 568 00:30:19.410 --> 00:30:21.110 number of children versus 569 00:30:21.110 --> 00:30:22.360 the predicted number of children. 570 00:30:22.360 --> 00:30:27.160 So that ratio is calculated to determine the risk 571 00:30:27.160 --> 00:30:29.060 of experiencing an outcome in a state. 572 00:30:30.160 --> 00:30:32.380 But you should look at, 573 00:30:32.380 --> 00:30:33.640 if you have questions about Risk-Standardized Performance, 574 00:30:33.640 --> 00:30:35.880 you should access the resources available 575 00:30:35.880 --> 00:30:38.310 on the Capacity Building Center site. 576 00:30:38.310 --> 00:30:39.990 There is, I'm trying to remember, 577 00:30:39.990 --> 00:30:42.000 and Elizabeth, maybe you can help me with that? 578 00:30:42.000 --> 00:30:44.540 I think there's a 2-page doc, 579 00:30:44.540 --> 00:30:45.777 2-page document that explains it, 580 00:30:45.777 --> 00:30:47.743 and then a separate FAQ? 581 00:30:49.430 --> 00:30:50.394 - [Elizabeth] That explains 582 00:30:50.394 --> 00:30:53.490 Risk-Standardized Performance in more detail? 583 00:30:53.490 --> 00:30:55.700 - Yes. - Yes, yes. 584 00:30:55.700 --> 00:30:58.710 There's FAQs on the Capacity Building Center for States. 585 00:30:58.710 --> 00:31:01.890 And also if you log in to the CFSR portal 586 00:31:01.890 --> 00:31:04.840 and the Online Monitoring System that provides visualization 587 00:31:04.840 --> 00:31:06.840 for the statewide data indicators, 588 00:31:06.840 --> 00:31:11.170 you'll also see more information about that. 589 00:31:11.170 --> 00:31:14.940 And TB #13 includes the data dictionary, 590 00:31:14.940 --> 00:31:16.770 and there's a column in the data dictionary 591 00:31:16.770 --> 00:31:20.820 that identifies the factors that are risk- 592 00:31:20.820 --> 00:31:23.053 adjusted by indicator. 593 00:31:26.270 --> 00:31:29.610 - [Roger] Thank you. Okay. 594 00:31:29.610 --> 00:31:31.420 Another question is when will the data profile 595 00:31:31.420 --> 00:31:34.460 for February 2022 be released? 596 00:31:34.460 --> 00:31:38.260 So that is the most recent upcoming data profile, 597 00:31:38.260 --> 00:31:43.260 which would still really apply to Round 3, and it will, 598 00:31:44.170 --> 00:31:49.170 right, it would not have the new indicator for re-entry 599 00:31:49.270 --> 00:31:51.870 or the updated indicator for permanency for entries. 600 00:31:52.800 --> 00:31:54.210 That is, it's forthcoming, I believe, 601 00:31:54.210 --> 00:31:56.721 within the next week or so. 602 00:31:56.721 --> 00:31:59.403 I think it's going through some final clearance processes. 603 00:32:01.180 --> 00:32:06.050 Another question is, is "statistically worse" defined? 604 00:32:06.050 --> 00:32:09.420 So we define "statistically worse" as using a 95% 605 00:32:09.420 --> 00:32:12.793 confidence interval for Risk-Standardized Performance. 606 00:32:14.555 --> 00:32:16.770 And that is how we define statistically worse, 607 00:32:16.770 --> 00:32:17.923 no different, or better. 608 00:32:21.164 --> 00:32:24.900 And then another question: Will Technical Bulletin #13 609 00:32:24.900 --> 00:32:27.740 be re-released with modifications? 610 00:32:27.740 --> 00:32:29.950 And can you estimate the new release date? 611 00:32:30.940 --> 00:32:34.190 We are updating Technical Bulletin #13 612 00:32:34.190 --> 00:32:36.017 with the national performance values 613 00:32:36.017 --> 00:32:37.713 and some other updates. 614 00:32:39.090 --> 00:32:42.640 We are planning that for spring 2022. 615 00:32:42.640 --> 00:32:46.840 I don't really have a more precise date than that, 616 00:32:46.840 --> 00:32:49.040 but it is coming within the next few months. 617 00:32:54.382 --> 00:32:56.968 - [Elizabeth] A few more questions coming your way, Roger. 618 00:32:56.968 --> 00:32:58.568 - [Roger] I see. Let's see here. 619 00:33:01.456 --> 00:33:03.220 So there's another question. 620 00:33:03.220 --> 00:33:05.010 The question about the SPSS code. 621 00:33:05.010 --> 00:33:09.220 So the question is if the SPSS code requires Python. 622 00:33:09.220 --> 00:33:14.220 What version of Python will the revised SPSS code use? 623 00:33:15.520 --> 00:33:18.731 So think it's a little tricky. 624 00:33:18.731 --> 00:33:23.731 So the syntax for Round 3 was written based 625 00:33:24.770 --> 00:33:29.593 on the SPSS version that used Python version 2, 626 00:33:30.910 --> 00:33:35.823 and then later SPSS updated to use Python 3. 627 00:33:37.920 --> 00:33:42.820 We released - available to the Capacity Building Center has 628 00:33:42.820 --> 00:33:45.400 a guide how to use the syntax, 629 00:33:45.400 --> 00:33:48.380 and they updated that with information 630 00:33:48.380 --> 00:33:51.130 on how to change the syntax 631 00:33:51.130 --> 00:33:56.090 if your SPSS version uses Python 3. Going forward, 632 00:33:56.090 --> 00:33:59.700 for Round 4, the syntax will be based around the SPSS 633 00:33:59.700 --> 00:34:02.440 version that uses Python 3. 634 00:34:02.440 --> 00:34:06.460 So the guides on how to use syntax will include 635 00:34:06.460 --> 00:34:09.680 now information for if you have an older version 636 00:34:09.680 --> 00:34:13.270 of SPSS which uses Python 2, how to change the syntax 637 00:34:13.270 --> 00:34:14.823 to use Python 2. 638 00:34:18.950 --> 00:34:20.850 Another question, could you please explain 639 00:34:20.850 --> 00:34:21.730 data quality limit 640 00:34:21.730 --> 00:34:24.410 percentages that are given on page 4 641 00:34:24.410 --> 00:34:28.223 of the data profile document published in August 2021? 642 00:34:29.610 --> 00:34:33.133 I don't have a copy of that easily available to me. 643 00:34:35.330 --> 00:34:37.401 I guess if you could explain more, 644 00:34:37.401 --> 00:34:40.350 by data quality limit percentages, are you referring 645 00:34:40.350 --> 00:34:43.470 to - for each data quality check, 646 00:34:43.470 --> 00:34:48.470 there is a limit to what the sort of requirement is, so, 647 00:34:49.010 --> 00:34:51.220 and I can't pull them up right now and I can't 648 00:34:51.220 --> 00:34:52.750 remember exactly what any of them are, but say, 649 00:34:52.750 --> 00:34:55.700 for example, there's a check for AFCARS IDs 650 00:34:55.700 --> 00:34:57.210 matching across periods. 651 00:34:57.210 --> 00:35:00.520 I think that is set at like 40%. 652 00:35:00.520 --> 00:35:04.330 And so if you have less than 40%, for example, 653 00:35:04.330 --> 00:35:09.330 then for that AFCARS period that is in question, 654 00:35:10.610 --> 00:35:13.460 you fail to meet the minimum requirement in the check 655 00:35:13.460 --> 00:35:17.093 and therefore that data quality check is failed. 656 00:35:21.730 --> 00:35:24.340 - [Elizabeth] So, Roger, I think he's referring to, 657 00:35:24.340 --> 00:35:26.740 as you mentioned, page 4 of the data profile. 658 00:35:27.620 --> 00:35:31.320 So all the, and he's mentioning the August 2021 profile. 659 00:35:31.320 --> 00:35:33.000 So yes, the limits are there. 660 00:35:33.000 --> 00:35:36.381 And limits were set ahead of Round 3, based 661 00:35:36.381 --> 00:35:40.443 on what would be a reasonable amount, 662 00:35:40.443 --> 00:35:45.443 or expected amount, of records that if it was outside, 663 00:35:45.810 --> 00:35:49.210 that limit would raise concerns about the quality 664 00:35:49.210 --> 00:35:50.043 of the data. 665 00:35:52.320 --> 00:35:53.153 - [Roger] Yes. 666 00:35:53.153 --> 00:35:54.960 Each, in each, there are, 667 00:35:54.960 --> 00:35:56.930 there are several data quality checks 668 00:35:56.930 --> 00:35:57.763 that are listed there. 669 00:35:57.763 --> 00:35:59.640 They're the same across every profile. 670 00:35:59.640 --> 00:36:02.140 So it's always the same quality checks. 671 00:36:03.620 --> 00:36:06.184 And the limits are given there. 672 00:36:06.184 --> 00:36:09.710 And then there's an indication of what a state's value is, 673 00:36:09.710 --> 00:36:12.550 so the state's percentage is there. 674 00:36:12.550 --> 00:36:15.760 And then that is color-coded to indicate 675 00:36:15.760 --> 00:36:17.793 whether or not the state's failed. 676 00:36:19.500 --> 00:36:21.470 Again, I think that while 677 00:36:21.470 --> 00:36:22.890 it's certainly important to know what those data 678 00:36:22.890 --> 00:36:23.723 quality checks are, right? 679 00:36:23.723 --> 00:36:24.920 It's also important to keep in mind 680 00:36:24.920 --> 00:36:29.440 that when you might see a single failure in a single 681 00:36:29.440 --> 00:36:30.727 time period in a data quality check, 682 00:36:30.727 --> 00:36:34.580 but that can impact multiple indicators across 683 00:36:34.580 --> 00:36:35.773 multiple time periods. 684 00:36:38.843 --> 00:36:40.453 - [Elizabeth] And TB #13 685 00:36:41.496 --> 00:36:43.350 includes the data dictionary as an appendix, 686 00:36:43.350 --> 00:36:46.850 and within the data dictionary is the description 687 00:36:46.850 --> 00:36:48.730 of each of those data indicators, 688 00:36:48.730 --> 00:36:50.470 and including what's counted, 689 00:36:50.470 --> 00:36:51.920 how it's calculated for the numerator, 690 00:36:51.920 --> 00:36:55.253 denominator, and the limits that's set. 691 00:36:58.560 --> 00:37:00.810 The good news is there's not that many states 692 00:37:00.810 --> 00:37:03.224 that are exceeding the data quality limits 693 00:37:03.224 --> 00:37:05.053 for the indicators. 694 00:37:13.401 --> 00:37:16.283 - [Roger] That seems to be all of the questions for now. 695 00:37:17.264 --> 00:37:19.210 - [Elizabeth] There's a few more, Roger. 696 00:37:19.210 --> 00:37:20.650 - Oh, there are. - Yeah. 697 00:37:20.650 --> 00:37:23.103 - [Roger] Okay. Oh, here we go. 698 00:37:25.928 --> 00:37:26.860 So the question asked, 699 00:37:26.860 --> 00:37:28.410 somebody asked if I can go back to the slide 700 00:37:28.410 --> 00:37:31.013 with the links to the resources, it is here. 701 00:37:32.650 --> 00:37:35.380 I believe this presentation is available or will 702 00:37:35.380 --> 00:37:37.250 be available for download. 703 00:37:37.250 --> 00:37:39.410 And several of these are clickable. 704 00:37:39.410 --> 00:37:42.960 So Technical Bulletin #13, Capacity Building Centers site, 705 00:37:42.960 --> 00:37:45.363 and the Information Portal are all clickable, 706 00:37:45.363 --> 00:37:48.533 will take you to those resources. 707 00:37:51.870 --> 00:37:56.180 - Okay. And I see this last question about the 2023 APSR. 708 00:37:56.180 --> 00:37:57.410 There is a requirement 709 00:37:59.440 --> 00:38:02.010 to provide an analysis of the state data indicators. 710 00:38:02.010 --> 00:38:04.690 Is the expectation that we use the new data indicator 711 00:38:04.690 --> 00:38:07.150 you plan to release in the spring? 712 00:38:07.150 --> 00:38:11.067 I don't think so. I think the 2023 APSR... 713 00:38:11.980 --> 00:38:14.773 Is that due in June of this year, June? 714 00:38:16.490 --> 00:38:17.800 - [Elizabeth] That's my understanding, 715 00:38:17.800 --> 00:38:19.810 the first draft. - So, yeah. 716 00:38:19.810 --> 00:38:24.810 So you should use the most recent data indicators 717 00:38:24.890 --> 00:38:26.490 that are available to you at the time 718 00:38:26.490 --> 00:38:27.450 that you're developing that, 719 00:38:27.450 --> 00:38:30.230 and that would probably be the February 2022 data 720 00:38:30.230 --> 00:38:32.890 that are being, profiles that will be released, 721 00:38:32.890 --> 00:38:37.890 as Roger said, imminently, in the coming week or so, 722 00:38:38.930 --> 00:38:39.830 I'll say week. 723 00:38:46.921 --> 00:38:47.754 - [Elizabeth] Jennifer, a couple questions 724 00:38:47.754 --> 00:38:48.930 about the availability 725 00:38:48.930 --> 00:38:51.980 of these presentations? - Right. 726 00:38:51.980 --> 00:38:54.380 The first presentation, 727 00:38:54.380 --> 00:38:58.380 I do not think it's been uploaded to the portal yet, 728 00:38:58.380 --> 00:39:01.940 but I think that it will be reasonably soon. 729 00:39:01.940 --> 00:39:06.940 And I think we can send out a notice to states, 730 00:39:08.780 --> 00:39:11.560 especially, we can, 731 00:39:11.560 --> 00:39:14.240 we can send out a notice on the listserv, 732 00:39:14.240 --> 00:39:17.610 through the OMS mailing list, or both, to make you aware 733 00:39:17.610 --> 00:39:19.053 of when they are posted. 734 00:39:24.540 --> 00:39:25.373 Okay. 735 00:39:25.373 --> 00:39:27.483 Are there any other questions? 736 00:39:29.989 --> 00:39:31.223 I'm not seeing any, 737 00:39:33.350 --> 00:39:38.340 so should we turn this meeting over, 738 00:39:39.290 --> 00:39:40.970 the next section, 739 00:39:40.970 --> 00:39:45.113 to go to the next section with Sylvia and Tammy? 740 00:39:49.710 --> 00:39:50.877 - Great. Thank you. 741 00:39:51.740 --> 00:39:54.150 My name is Sylvia Kim with the CFSR Unit, 742 00:39:54.150 --> 00:39:57.180 and I am also a member of the MASC team, 743 00:39:57.180 --> 00:40:01.060 as Jennifer referenced earlier in our opening remarks. 744 00:40:01.060 --> 00:40:04.050 Today, I will go through the CFSR case review framework 745 00:40:04.050 --> 00:40:04.904 in the next few slides, 746 00:40:04.904 --> 00:40:08.703 and the PIP measurement framework will be covered next week. 747 00:40:13.560 --> 00:40:16.270 First, let's cover the CFSR timeline. 748 00:40:16.270 --> 00:40:19.560 The timeline is the same for all the rounds of CFSRs. 749 00:40:19.560 --> 00:40:22.390 The most recent data profile will be transmitted 750 00:40:22.390 --> 00:40:25.480 to states approximately 6 months prior to the start 751 00:40:25.480 --> 00:40:27.030 of the onsite review, 752 00:40:27.030 --> 00:40:29.620 along with the Statewide Assessment template, 753 00:40:29.620 --> 00:40:31.910 and that will signal that the state should start 754 00:40:31.910 --> 00:40:34.313 the statewide assessment if you haven't already. 755 00:40:39.330 --> 00:40:42.020 Okay. Case review framework. 756 00:40:42.020 --> 00:40:45.690 As in Round 3, states may propose to have either 757 00:40:45.690 --> 00:40:48.560 a CB-Led Review, which is an onsite review 758 00:40:48.560 --> 00:40:51.240 that occurs over a week with a federal team, 759 00:40:51.240 --> 00:40:52.880 or a State-Led Review, 760 00:40:52.880 --> 00:40:55.730 which is conducted by the state's own case review team 761 00:40:55.730 --> 00:40:59.800 and can occur over a maximum 6-month period. 762 00:40:59.800 --> 00:41:01.240 For both types of reviews, 763 00:41:01.240 --> 00:41:03.740 a minimum of three sites will be selected, 764 00:41:03.740 --> 00:41:07.300 which includes the largest metropolitan area. 765 00:41:07.300 --> 00:41:10.143 I will go over the site selection process a little later. 766 00:41:10.143 --> 00:41:12.880 The sample size will be the same as Round 3, 767 00:41:12.880 --> 00:41:15.620 with a minimum of 65 cases. 768 00:41:15.620 --> 00:41:18.030 For states choosing to do their own reviews, 769 00:41:18.030 --> 00:41:19.957 the sample size can be larger. 770 00:41:19.957 --> 00:41:22.670 And as far as sampling methodology is concerned, 771 00:41:22.670 --> 00:41:26.000 we want to simplify stratification for Round 4. 772 00:41:26.000 --> 00:41:29.310 In Round 3, many states stratified by geography, 773 00:41:29.310 --> 00:41:31.980 such as regions, districts, and counties. 774 00:41:31.980 --> 00:41:33.860 However, there are also some states that had 775 00:41:33.860 --> 00:41:37.400 multiple strata on top of geography and case type such 776 00:41:37.400 --> 00:41:40.030 as permanency goals and even age of the child. 777 00:41:40.030 --> 00:41:43.307 When we have small samples, and most states chose 778 00:41:43.307 --> 00:41:46.580 to review the minimum number of 65 cases, 779 00:41:46.580 --> 00:41:47.990 that's a lot of strata, 780 00:41:47.990 --> 00:41:51.710 which lessens to some degree the randomness of the sample. 781 00:41:51.710 --> 00:41:53.320 With that being said, 782 00:41:53.320 --> 00:41:55.710 we do want to be open to state proposals, 783 00:41:55.710 --> 00:41:58.320 we want to listen to your recommendations and rationale 784 00:41:58.320 --> 00:42:00.120 for stratifying the sample, 785 00:42:00.120 --> 00:42:03.060 especially if your state is going to review a large 786 00:42:03.060 --> 00:42:04.450 number of cases. 787 00:42:04.450 --> 00:42:07.300 We also recognize that when we limit stratification, 788 00:42:07.300 --> 00:42:10.430 we then may need to negotiate with the state the use 789 00:42:10.430 --> 00:42:11.780 of setting caps. 790 00:42:11.780 --> 00:42:13.590 So cases from the metro site, 791 00:42:13.590 --> 00:42:16.430 or maybe an in-home services case type like 792 00:42:16.430 --> 00:42:19.236 alternative response do not overly dominate 793 00:42:19.236 --> 00:42:21.430 the sample frames. 794 00:42:21.430 --> 00:42:23.430 Before we move on to the next slide, 795 00:42:23.430 --> 00:42:26.060 I want to answer one of the questions we received 796 00:42:26.060 --> 00:42:28.140 during the February 23rd national call 797 00:42:28.140 --> 00:42:30.040 with the Associate Commissioner 798 00:42:30.040 --> 00:42:32.400 Aysha Schomburg and Jennifer Haight. 799 00:42:32.400 --> 00:42:35.421 One jurisdiction asked whether they should sample cases 800 00:42:35.421 --> 00:42:39.020 that represent their population by race and gender, 801 00:42:39.020 --> 00:42:42.920 as well as by case type of foster care or in-home. 802 00:42:42.920 --> 00:42:45.930 While it's true that we want states to disaggregate 803 00:42:45.930 --> 00:42:50.170 data and dig deeper into the data to understand how race, 804 00:42:50.170 --> 00:42:53.700 ethnicity, gender, or even age may be impacting 805 00:42:53.700 --> 00:42:56.620 a state's child welfare system, for the purposes of doing 806 00:42:56.620 --> 00:42:59.680 the statewide assessment, we don't expect 807 00:42:59.680 --> 00:43:03.330 or encourage states to purposefully stratify case 808 00:43:03.330 --> 00:43:05.159 samples by these subgroups. 809 00:43:05.159 --> 00:43:06.640 As I mentioned earlier, 810 00:43:06.640 --> 00:43:09.710 we really do want to simplify the sampling methodology 811 00:43:09.710 --> 00:43:11.200 as much as possible, 812 00:43:11.200 --> 00:43:14.400 and we encourage states to rely on the distribution 813 00:43:14.400 --> 00:43:17.200 of a random sample that should, in effect, 814 00:43:17.200 --> 00:43:18.833 reflect the case population. 815 00:43:22.050 --> 00:43:24.423 So let's cover the site selection process. 816 00:43:27.460 --> 00:43:30.194 The onsite review phase includes an examination 817 00:43:30.194 --> 00:43:33.110 of the experiences of children and families served 818 00:43:33.110 --> 00:43:34.950 by the child welfare system. 819 00:43:34.950 --> 00:43:37.380 Information is collected by reviewing a sample 820 00:43:37.380 --> 00:43:40.010 of cases and conducting interviews with case 821 00:43:40.010 --> 00:43:42.970 participants and is used to determine whether a state 822 00:43:42.970 --> 00:43:45.130 is operating in substantial conformity 823 00:43:45.130 --> 00:43:49.710 with federal requirements pertaining to the seven child 824 00:43:49.710 --> 00:43:53.070 and family outcomes around Safety, Permanency, and Well-Being. 825 00:43:53.070 --> 00:43:55.810 Cases are reviewed at a selection of sites 826 00:43:55.810 --> 00:43:58.580 that reflect a cross-section of the state, 827 00:43:58.580 --> 00:44:01.080 including rural and suburban areas. 828 00:44:01.080 --> 00:44:03.510 Case reviews provide evidence of state child 829 00:44:03.510 --> 00:44:05.760 welfare practice, system performance, 830 00:44:05.760 --> 00:44:08.152 as well as opportunities to further analyze 831 00:44:08.152 --> 00:44:11.150 system strengths and areas needing improvement. 832 00:44:11.150 --> 00:44:12.940 For CB-Led Reviews, 833 00:44:12.940 --> 00:44:16.700 three sites will be selected, and for State-Led Reviews, 834 00:44:16.700 --> 00:44:19.630 the number of review sites will be negotiated 835 00:44:19.630 --> 00:44:24.470 with a state when a statewide random sample is not used. 836 00:44:24.470 --> 00:44:26.750 I want to stress that the selection of sites 837 00:44:26.750 --> 00:44:29.070 is a collaborative process between states 838 00:44:29.070 --> 00:44:32.220 and CB, with CB having final approval. 839 00:44:32.220 --> 00:44:35.270 States and CB use state data and information 840 00:44:35.270 --> 00:44:39.290 to explore prospective sites and child welfare system 841 00:44:39.290 --> 00:44:42.411 and practice dynamics to be examined during the onsite review. 842 00:44:42.411 --> 00:44:45.360 We've developed a set of guidelines for states to use 843 00:44:45.360 --> 00:44:47.290 in selecting sites that will hopefully provide 844 00:44:47.290 --> 00:44:49.987 maximum flexibility to determine the most 845 00:44:49.987 --> 00:44:52.893 appropriate review sites, given each state's 846 00:44:52.893 --> 00:44:57.160 unique characteristics, practices, systems, and population. 847 00:44:57.160 --> 00:44:59.490 In the next slide, we'll look at some of the factors 848 00:44:59.490 --> 00:45:02.716 states should consider when thinking about site selection. 849 00:45:02.716 --> 00:45:05.510 I should also mention that this guide, 850 00:45:05.510 --> 00:45:09.480 it is titled CFSR Round 4 Site Selection Proposals, 851 00:45:09.480 --> 00:45:11.650 is available at the CFSR portal, 852 00:45:11.650 --> 00:45:14.190 along with the TB #13 and the other 853 00:45:14.190 --> 00:45:17.433 resources referenced by Roger and Jennifer earlier. 854 00:45:18.610 --> 00:45:20.320 - [Elizabeth] Sylvia, we have a question 855 00:45:20.320 --> 00:45:22.440 from Doug in Washington, what constitutes 856 00:45:22.440 --> 00:45:23.973 a statewide random sample? 857 00:45:25.470 --> 00:45:28.034 - Statewide random sample would be if you were to draw 858 00:45:28.034 --> 00:45:32.290 a sample of children and families from the entire 859 00:45:32.290 --> 00:45:35.730 state and randomize and pull a number of cases 860 00:45:35.730 --> 00:45:36.563 that you were being, 861 00:45:36.563 --> 00:45:39.870 so you're not stratifying by region or district or site, 862 00:45:39.870 --> 00:45:42.819 but you're pulling the entire population, 863 00:45:42.819 --> 00:45:46.420 the sample from the entire population being served 864 00:45:46.420 --> 00:45:47.340 in the state. 865 00:45:47.340 --> 00:45:50.470 I don't know if other MASC members have anything to add 866 00:45:50.470 --> 00:45:52.280 to that, but that's what I would describe 867 00:45:52.280 --> 00:45:54.353 it as the statewide random sample. 868 00:45:58.180 --> 00:46:01.223 Did that answer the question? Are there any other questions? 869 00:46:04.610 --> 00:46:06.463 Okay, let me move on. 870 00:46:07.934 --> 00:46:10.370 Factors to consider when selecting sites 871 00:46:10.370 --> 00:46:13.391 for the onsite review will be similar as in Round 3. 872 00:46:13.391 --> 00:46:16.640 States will need to submit a site selection 873 00:46:16.640 --> 00:46:19.200 proposal that addresses the seven factors 874 00:46:19.200 --> 00:46:21.370 that are displayed on the slide. 875 00:46:21.370 --> 00:46:23.890 The first factor is in regard to the largest 876 00:46:23.890 --> 00:46:26.070 metropolitan area in the state. 877 00:46:26.070 --> 00:46:29.090 This is a required site for the CFSR onsite review 878 00:46:29.090 --> 00:46:30.231 per regulations. 879 00:46:30.231 --> 00:46:33.711 In some states, if two or more of the largest cities 880 00:46:33.711 --> 00:46:36.220 are identified as metro areas, 881 00:46:36.220 --> 00:46:38.950 we will ask states to provide the supporting evidence 882 00:46:38.950 --> 00:46:42.410 and states' recommendation for the city to be used 883 00:46:42.410 --> 00:46:44.490 as the metropolitan site. 884 00:46:44.490 --> 00:46:46.950 As needed, we'll take a closer look at the state 885 00:46:46.950 --> 00:46:50.019 Census child population counts and projections as part 886 00:46:50.019 --> 00:46:53.230 of the data we will use during negotiations. 887 00:46:53.230 --> 00:46:55.930 The second factor to consider is the diversity 888 00:46:55.930 --> 00:46:58.950 and characteristics of the population in the state. 889 00:46:58.950 --> 00:47:02.130 In the site selection proposal, states should describe 890 00:47:02.130 --> 00:47:05.160 the racial and ethnic composition of the state 891 00:47:05.160 --> 00:47:07.850 and the children and families who are served. 892 00:47:07.850 --> 00:47:09.360 States should provide information 893 00:47:09.360 --> 00:47:12.940 to show whether there is or [is] not evidence 894 00:47:12.940 --> 00:47:15.770 of potential disproportionality in the system 895 00:47:15.770 --> 00:47:18.830 and disparities and outcomes for persons of color 896 00:47:18.830 --> 00:47:22.092 and others who have been historically underserved, 897 00:47:22.092 --> 00:47:25.700 marginalized, and adversely affected by poverty 898 00:47:25.700 --> 00:47:28.480 and inequality in the child welfare system. 899 00:47:28.480 --> 00:47:30.160 Aysha Schomburg addressed that 900 00:47:30.160 --> 00:47:32.500 in the February 23rd call. 901 00:47:32.500 --> 00:47:34.700 And that is what we would expect the states 902 00:47:34.700 --> 00:47:39.490 to actually provide in the site selection proposal. 903 00:47:39.490 --> 00:47:41.010 The state should also identify 904 00:47:41.010 --> 00:47:44.010 American Indian or Alaskan Tribes in the state 905 00:47:44.010 --> 00:47:46.990 and the degree of Tribal involvement in the provision 906 00:47:46.990 --> 00:47:48.363 of child welfare services. 907 00:47:49.240 --> 00:47:51.790 The proposal should address whether the combination 908 00:47:51.790 --> 00:47:55.410 of proposed sites reflects the key data I just spoke about, 909 00:47:55.410 --> 00:47:58.810 and if not, the rationale for proposing sites that do 910 00:47:58.810 --> 00:48:01.130 not reflect the state data. 911 00:48:01.130 --> 00:48:03.560 The third factor is whether there's a sufficient number 912 00:48:03.560 --> 00:48:06.620 of foster care and in-home cases in the proposed sites 913 00:48:06.620 --> 00:48:08.493 to support random sampling. 914 00:48:08.493 --> 00:48:12.120 A review site should have at least 46 times more 915 00:48:12.120 --> 00:48:15.160 cases than the number of cases scheduled for a review 916 00:48:15.160 --> 00:48:16.760 in that site. 917 00:48:16.760 --> 00:48:19.151 We'll ask states to describe the in-home services 918 00:48:19.151 --> 00:48:22.390 and the foster care populations in the proposed sites, 919 00:48:22.390 --> 00:48:24.102 and that would include the types of programs 920 00:48:24.102 --> 00:48:27.363 and cases that are counted in each case type. 921 00:48:27.363 --> 00:48:30.680 We also recommend that you identify the locality 922 00:48:30.680 --> 00:48:33.577 or jurisdiction in the state with the largest total 923 00:48:33.577 --> 00:48:36.773 case population as a possible review site. 924 00:48:37.970 --> 00:48:40.647 Another factor we consider is child welfare practice 925 00:48:40.647 --> 00:48:41.930 and system performance. 926 00:48:41.930 --> 00:48:45.180 States should describe whether the combination 927 00:48:45.180 --> 00:48:48.790 of proposed sites is similar to statewide performance 928 00:48:48.790 --> 00:48:52.030 in various data sets, like the statewide 929 00:48:52.030 --> 00:48:55.340 data indicators, the case review performance data, 930 00:48:55.340 --> 00:48:56.773 and other key measures. 931 00:48:57.840 --> 00:49:00.925 States should use a combination of data sources, including 932 00:49:00.925 --> 00:49:03.540 but not limited to the quantitative 933 00:49:03.540 --> 00:49:06.373 and the qualitative data to identify potential sites. 934 00:49:07.610 --> 00:49:10.725 A fifth factor we ask states to consider are 935 00:49:10.725 --> 00:49:13.450 the urban and rural characteristics. 936 00:49:13.450 --> 00:49:15.684 The site selection proposal should identify 937 00:49:15.684 --> 00:49:20.270 whether proposed sites are urban or rural in nature, 938 00:49:20.270 --> 00:49:23.110 and whether services or resources needed by families 939 00:49:23.110 --> 00:49:25.623 are readily available or accessible. 940 00:49:26.790 --> 00:49:29.680 Next, states should assess potential sites 941 00:49:29.680 --> 00:49:32.050 for new programs and initiatives. 942 00:49:32.050 --> 00:49:34.540 Are the sites implementing innovative practices 943 00:49:34.540 --> 00:49:37.410 or programs that are having a positive impact, and would 944 00:49:37.410 --> 00:49:40.490 it benefit the state to examine them through the review? 945 00:49:40.490 --> 00:49:41.880 And finally, but not least, 946 00:49:41.880 --> 00:49:44.290 states should consider the level of community 947 00:49:44.290 --> 00:49:46.420 partnerships in the potential sites. 948 00:49:46.420 --> 00:49:49.740 Are there strong partnerships in these sites among youth, 949 00:49:49.740 --> 00:49:52.920 families, Tribes, legal- judicial communities, 950 00:49:52.920 --> 00:49:55.160 that will help provide insight that would benefit 951 00:49:55.160 --> 00:49:57.480 the child welfare system statewide? 952 00:49:57.480 --> 00:49:59.110 So those are the seven factors 953 00:50:01.552 --> 00:50:03.240 that states should consider 954 00:50:03.240 --> 00:50:05.203 as you think about site selection. 955 00:50:09.500 --> 00:50:11.880 So now let's go over some changes we made 956 00:50:11.880 --> 00:50:13.870 to the case review populations. 957 00:50:13.870 --> 00:50:16.040 These changes will apply to both 958 00:50:16.040 --> 00:50:18.530 the CFSR and PIP measurements. 959 00:50:18.530 --> 00:50:22.320 First, the foster care case populations did not change. 960 00:50:22.320 --> 00:50:25.330 They will be the same for Round 4 as prior rounds. 961 00:50:25.330 --> 00:50:28.810 We will continue to exclude trial home visit cases 962 00:50:28.810 --> 00:50:31.770 and cases for youth who are older than 18. 963 00:50:31.770 --> 00:50:34.172 For in-home services cases, 964 00:50:34.172 --> 00:50:37.640 we are going to continue to require that the case be 965 00:50:37.640 --> 00:50:41.020 open for a minimum of 45 consecutive days 966 00:50:41.020 --> 00:50:42.600 during the sampling period. 967 00:50:42.600 --> 00:50:45.610 We will also continue to include in-home cases 968 00:50:45.610 --> 00:50:49.210 that are open for ongoing services or case management. 969 00:50:49.210 --> 00:50:52.467 However, a change for Round 4 is that we will not 970 00:50:52.467 --> 00:50:57.062 include CPS reports that only received an investigation 971 00:50:57.062 --> 00:50:59.103 or assessment response. 972 00:51:01.570 --> 00:51:03.920 So I'm going to skip to the third sub-bullet 973 00:51:03.920 --> 00:51:06.350 under the in-home services population 974 00:51:06.350 --> 00:51:08.700 before discussing the second sub-bullet. 975 00:51:08.700 --> 00:51:11.450 When we look at the array of non-foster care cases 976 00:51:11.450 --> 00:51:14.500 that are part of the state's child welfare continuum, 977 00:51:14.500 --> 00:51:17.910 we will include cases that were open for at least 45 978 00:51:17.910 --> 00:51:22.150 days and were funded with federal title IV-B dollars 979 00:51:22.150 --> 00:51:26.090 and were part of the services included in the CFSP. 980 00:51:26.090 --> 00:51:27.990 The state NCB will engage 981 00:51:27.990 --> 00:51:30.630 in conversations about services provided 982 00:51:30.630 --> 00:51:32.910 under the state's IV-B, IV-E plan, 983 00:51:32.910 --> 00:51:35.530 which is also the CFSP, or the Child and Family 984 00:51:35.530 --> 00:51:39.500 Services Plan, to determine the in-home services 985 00:51:39.500 --> 00:51:42.690 case population that are subject to review 986 00:51:42.690 --> 00:51:43.883 through the CFSR. 987 00:51:46.690 --> 00:51:49.210 So let's talk about the second sub-bullet. 988 00:51:49.210 --> 00:51:51.940 The biggest change that we have for Round 4 989 00:51:51.940 --> 00:51:56.260 is the inclusion of children who are on a trial home 990 00:51:56.260 --> 00:51:59.480 visit in the in-home services case population. 991 00:51:59.480 --> 00:52:01.900 While cases with children on a trial home visit 992 00:52:01.900 --> 00:52:05.840 are included in the AFCARS reportable case population, 993 00:52:05.840 --> 00:52:08.320 these children who are placed at home are not 994 00:52:08.320 --> 00:52:10.370 technically in foster care. 995 00:52:10.370 --> 00:52:12.860 The definition in the regulation which defines 996 00:52:12.860 --> 00:52:15.117 what foster care is, quote unquote, 997 00:52:15.117 --> 00:52:17.900 "24-hour substitute care for children placed 998 00:52:17.900 --> 00:52:20.180 away from their parents or guardians, 999 00:52:20.180 --> 00:52:23.110 and for whom the title IV-E agency has placement 1000 00:52:23.110 --> 00:52:25.190 and care responsibility." 1001 00:52:25.190 --> 00:52:28.636 In Round 3, these cases were eliminated altogether 1002 00:52:28.636 --> 00:52:30.980 from the foster care sampling frame. 1003 00:52:30.980 --> 00:52:33.991 And for Round 4, we want to review these cases 1004 00:52:33.991 --> 00:52:36.830 since casework for children residing at home 1005 00:52:36.830 --> 00:52:39.200 under the placement and care responsibility 1006 00:52:39.200 --> 00:52:43.360 of the state is an important case work practice to assess. 1007 00:52:43.360 --> 00:52:45.784 We recognize that we'll need to work with the states 1008 00:52:45.784 --> 00:52:48.620 to identify and include these cases 1009 00:52:48.620 --> 00:52:50.910 in the in-home services sampling frame. 1010 00:52:50.910 --> 00:52:54.210 We're also developing, I think it's just cleared, 1011 00:52:54.210 --> 00:52:58.230 a guidance and instruction to provide to states 1012 00:52:58.230 --> 00:52:59.880 during the consultation process 1013 00:52:59.880 --> 00:53:04.600 with the MASC to help identify these cases 1014 00:53:04.600 --> 00:53:06.860 and to include them in the in-home 1015 00:53:06.860 --> 00:53:09.050 services case population. 1016 00:53:09.050 --> 00:53:10.950 Lastly, MASC will review, 1017 00:53:10.950 --> 00:53:14.287 verify accuracy of the foster care and in-home services 1018 00:53:14.287 --> 00:53:17.712 sampling frames, just like we did in Round 3, 1019 00:53:17.712 --> 00:53:21.700 as we move toward the onsite review process. 1020 00:53:21.700 --> 00:53:26.240 That covers the case review framework portion 1021 00:53:26.240 --> 00:53:27.943 of my presentation. 1022 00:53:28.940 --> 00:53:31.440 I will now turn this over to Tammy White, 1023 00:53:31.440 --> 00:53:34.190 unless there are questions. - A few questions about the... 1024 00:53:34.190 --> 00:53:35.023 - Okay. 1025 00:53:38.856 --> 00:53:40.110 - Do you see those? - If you let me scroll, 1026 00:53:40.110 --> 00:53:42.530 I'm going to scroll to the questions now. 1027 00:53:42.530 --> 00:53:45.770 I think we answered Doug's question, correct? 1028 00:53:45.770 --> 00:53:48.263 - [Elizabeth] Yep. 1029 00:53:51.060 --> 00:53:54.540 - April Simmons, for the in-home services population, 1030 00:53:54.540 --> 00:53:58.383 will the change include whether the case is open or closed? 1031 00:53:59.500 --> 00:54:00.480 Okay. 1032 00:54:00.480 --> 00:54:03.760 Meaning would a case be eligible if the family 1033 00:54:03.760 --> 00:54:07.260 was referred for services, but the case closed? 1034 00:54:07.260 --> 00:54:10.170 As long as the case is open for 45 days 1035 00:54:10.170 --> 00:54:14.210 during the sampling period, we will review those cases, 1036 00:54:14.210 --> 00:54:16.390 regardless of whether they're open or closed, 1037 00:54:16.390 --> 00:54:18.810 but the key issue is that they have to be open 1038 00:54:18.810 --> 00:54:22.673 for 45 consecutive days during the sampling period. 1039 00:54:23.900 --> 00:54:26.420 Does anyone else have anything else to add 1040 00:54:26.420 --> 00:54:29.253 to that in terms of these cases? 1041 00:54:32.480 --> 00:54:36.635 The next question from, I'm not sure, 1042 00:54:36.635 --> 00:54:38.250 I'm having a hard time, 1043 00:54:38.250 --> 00:54:39.970 these boxes are really small and I'm having 1044 00:54:39.970 --> 00:54:41.060 a hard time scrolling. 1045 00:54:41.060 --> 00:54:43.511 What if these THV children were in 1046 00:54:43.511 --> 00:54:47.700 out-of-home care for certain days during the review period? 1047 00:54:47.700 --> 00:54:49.060 That's a good question. 1048 00:54:49.060 --> 00:54:52.020 If, and so we'll be looking at foster children who come 1049 00:54:52.020 --> 00:54:53.320 up in the AFCARS spring. 1050 00:54:53.320 --> 00:54:54.545 Correct? 1051 00:54:54.545 --> 00:54:58.980 And so if these children are in 1052 00:54:58.980 --> 00:55:02.410 out-of-home care at any point during the sampling period 1053 00:55:02.410 --> 00:55:03.990 and the period under review, 1054 00:55:03.990 --> 00:55:05.830 these cases will be eliminated. 1055 00:55:05.830 --> 00:55:08.598 So it'll be just like any other in-home case. 1056 00:55:08.598 --> 00:55:10.490 We ask that any child, 1057 00:55:10.490 --> 00:55:13.230 or if a sibling enters foster care during the period 1058 00:55:13.230 --> 00:55:14.270 under review, 1059 00:55:14.270 --> 00:55:16.730 then we need to eliminate that case. 1060 00:55:16.730 --> 00:55:20.120 And that same condition will apply to these THV cases 1061 00:55:20.120 --> 00:55:23.123 once they're part of the in-home sampling frame. 1062 00:55:25.650 --> 00:55:26.483 Okay. 1063 00:55:26.483 --> 00:55:27.920 There's another question. 1064 00:55:27.920 --> 00:55:30.950 Is it correct that children who are having a trial 1065 00:55:30.950 --> 00:55:33.680 home visit at the start of the PUR that is less 1066 00:55:33.680 --> 00:55:37.560 than 45 days will not be eligible for either in-home 1067 00:55:37.560 --> 00:55:39.400 or out-of-home case review? 1068 00:55:39.400 --> 00:55:41.360 Yes, that is correct. 1069 00:55:41.360 --> 00:55:42.580 Once we move these children 1070 00:55:42.580 --> 00:55:45.300 over from the AFCARS or the foster care sampling 1071 00:55:45.300 --> 00:55:47.020 frame into the in-home, 1072 00:55:47.020 --> 00:55:50.110 same elimination criteria will apply, 1073 00:55:50.110 --> 00:55:53.010 which includes, if the case has been open for less 1074 00:55:53.010 --> 00:55:56.150 than 45 days during the sampling period and the period 1075 00:55:56.150 --> 00:55:59.713 under review, then that case will need to be eliminated. 1076 00:56:02.690 --> 00:56:04.170 There's another question. 1077 00:56:04.170 --> 00:56:08.620 If a family is assigned to an in-home unit case 1078 00:56:08.620 --> 00:56:11.640 manager for an additional assessment period, 1079 00:56:11.640 --> 00:56:15.290 but the family declined services and the case closes 1080 00:56:15.290 --> 00:56:18.610 with no services, is that case considered to have 1081 00:56:18.610 --> 00:56:22.563 only received CPS investigation, assessment, response? 1082 00:56:26.776 --> 00:56:29.510 The answer that comes to the top of my head is yes, 1083 00:56:29.510 --> 00:56:32.590 but I want to invite the other MASC members to make 1084 00:56:32.590 --> 00:56:35.050 sure that I am correct on this. 1085 00:56:35.050 --> 00:56:38.760 If the case only received an investigation 1086 00:56:38.760 --> 00:56:42.130 or an assessment, then yes, 1087 00:56:42.130 --> 00:56:44.750 that case will need to be eliminated from the case 1088 00:56:44.750 --> 00:56:46.960 sample even though the case may have been open for more 1089 00:56:46.960 --> 00:56:48.173 than 45 days. 1090 00:56:51.870 --> 00:56:52.703 Okay. 1091 00:56:52.703 --> 00:56:55.900 The next, another question is, if a family reserved, 1092 00:56:55.900 --> 00:56:56.850 oh, okay. 1093 00:56:56.850 --> 00:56:59.620 If a family received in-home services and agency 1094 00:56:59.620 --> 00:57:02.470 case management, but the services are not funded 1095 00:57:02.470 --> 00:57:06.350 by title IV-B, can you all tell, 1096 00:57:06.350 --> 00:57:07.626 I need my glasses, 1097 00:57:07.626 --> 00:57:10.593 is that case included in the in-home sample? 1098 00:57:14.370 --> 00:57:17.390 Oh, that is a very, it's a good question, 1099 00:57:17.390 --> 00:57:18.980 but it's a complicated question. 1100 00:57:18.980 --> 00:57:22.650 I think that we'll have to explore more information 1101 00:57:22.650 --> 00:57:24.730 and we'll have to talk to the state 1102 00:57:24.730 --> 00:57:27.420 about these types of cases before we can make 1103 00:57:27.420 --> 00:57:29.620 a decision one way or the other. 1104 00:57:29.620 --> 00:57:32.410 I don't think that I can answer just based on the fact 1105 00:57:32.410 --> 00:57:35.250 that they're not receiving IV-B, I think we'll have 1106 00:57:35.250 --> 00:57:38.320 to explore that more with a state around all 1107 00:57:38.320 --> 00:57:42.150 of these cases, non-foster care cases 1108 00:57:42.150 --> 00:57:44.670 as we discuss sampling with the state. 1109 00:57:44.670 --> 00:57:48.060 So I don't know that I can give a clear-cut answer, 1110 00:57:48.060 --> 00:57:50.290 but usually, but for the most part, 1111 00:57:50.290 --> 00:57:53.230 we are looking for cases that are funded by title IV-B. 1112 00:57:55.160 --> 00:57:56.990 Anyone else from MASC, or Jennifer? 1113 00:57:56.990 --> 00:57:59.873 Do you guys have any comments or input about that? 1114 00:58:01.610 --> 00:58:02.443 - No- - I think 1115 00:58:02.443 --> 00:58:03.790 that's largely right. 1116 00:58:03.790 --> 00:58:05.313 Sorry, go ahead. - Yeah. 1117 00:58:06.500 --> 00:58:07.500 - All right. Thanks. 1118 00:58:09.000 --> 00:58:10.380 Okay. Next question. 1119 00:58:10.380 --> 00:58:13.550 Does the CFSR site selection proposal lock a state 1120 00:58:13.550 --> 00:58:17.200 into where they conduct PIP measurement sites? 1121 00:58:17.200 --> 00:58:19.015 And that question is no. 1122 00:58:19.015 --> 00:58:22.670 The CFSR review is disconnected from the PIP 1123 00:58:22.670 --> 00:58:24.517 measurement in the sense that we, 1124 00:58:24.517 --> 00:58:26.310 and that is why we're making clear 1125 00:58:26.310 --> 00:58:28.980 that this presentation today is in regard 1126 00:58:28.980 --> 00:58:32.929 to the CFSR, and as far as the PIP is concerned, 1127 00:58:32.929 --> 00:58:34.780 it will be a disconnected process 1128 00:58:34.780 --> 00:58:37.120 in that when you are developing 1129 00:58:37.120 --> 00:58:39.029 your PIP and assessing your systems 1130 00:58:39.029 --> 00:58:42.900 and identifying strategies, there will be a different 1131 00:58:42.900 --> 00:58:46.881 set of PIP sites that you'll be able to select, 1132 00:58:46.881 --> 00:58:50.063 just like you do for the CFSR site selection, 1133 00:58:50.063 --> 00:58:54.093 that has to correspond to which areas that you want 1134 00:58:54.093 --> 00:58:56.010 to implement your strategies, 1135 00:58:56.010 --> 00:58:58.850 the kind of strategies that you want to implement 1136 00:58:58.850 --> 00:59:01.510 that will designate the PIP measurement sites. 1137 00:59:01.510 --> 00:59:05.570 So the simple answer is no to your question, 1138 00:59:05.570 --> 00:59:07.520 but we'll be working with you, again, 1139 00:59:07.520 --> 00:59:10.260 as you're working on your PIP to determine 1140 00:59:10.260 --> 00:59:13.480 which sites would be best suited for PIP measurement 1141 00:59:13.480 --> 00:59:14.583 when that time comes. 1142 00:59:17.820 --> 00:59:18.653 - [Elizabeth] Other questions will be about 1143 00:59:18.653 --> 00:59:19.513 trial home visits. 1144 00:59:20.530 --> 00:59:22.410 - Okay. These fonts are very, very small. 1145 00:59:22.410 --> 00:59:25.250 I may have to really whip out my glasses. 1146 00:59:25.250 --> 00:59:28.330 To clarify, the THV cases may be listed as foster 1147 00:59:28.330 --> 00:59:31.074 care cases in our local area systems. 1148 00:59:31.074 --> 00:59:32.640 But if they're in the home, 1149 00:59:32.640 --> 00:59:36.110 on a trial home visit for the entirety of the PUR, 1150 00:59:36.110 --> 00:59:40.303 is it considered, I'm sorry, I lost that. 1151 00:59:43.940 --> 00:59:46.460 I lost the question as I was scrolling. 1152 00:59:46.460 --> 00:59:47.420 - [Elizabeth] I'll read it to you, Sylvia. 1153 00:59:47.420 --> 00:59:48.610 The question is, 1154 00:59:48.610 --> 00:59:51.920 is it considered to be an in-home services case 1155 00:59:51.920 --> 00:59:54.240 and will it be reviewed as one? 1156 00:59:54.240 --> 00:59:56.350 - Yes. - So if, yeah. Okay. 1157 00:59:56.350 --> 00:59:59.097 - Yes, the answer is yes. - Okay. 1158 00:59:59.097 --> 01:00:02.950 And the next one is, there was an audio issue, 1159 01:00:02.950 --> 01:00:06.170 but did I hear correctly that alternative response 1160 01:00:06.170 --> 01:00:08.820 cases that only receive an assessment and are 1161 01:00:08.820 --> 01:00:10.623 excluded from the in-home sample, 1162 01:00:11.810 --> 01:00:14.860 but if at the assessment the worker identifies a need 1163 01:00:14.860 --> 01:00:17.450 for services and refers the family to services, 1164 01:00:17.450 --> 01:00:20.352 then those cases would come into the sample as long 1165 01:00:20.352 --> 01:00:23.420 as the case has been open for 45 days? 1166 01:00:23.420 --> 01:00:26.620 - Yes, that's correct. - Okay. 1167 01:00:26.620 --> 01:00:31.620 And then there's a correction to a prior question, 1168 01:00:31.670 --> 01:00:33.670 but I think we understood the question and I think 1169 01:00:33.670 --> 01:00:36.423 we answered if we didn't, can you please let us know? 1170 01:00:41.251 --> 01:00:45.584 And that, as far as I can see, is the last question. 1171 01:00:46.640 --> 01:00:48.120 Oh, thank you. Thanks for confirming. 1172 01:00:48.120 --> 01:00:49.803 We did answer. Okay. 1173 01:00:50.780 --> 01:00:51.650 - Great. Thank you. 1174 01:00:51.650 --> 01:00:53.600 And now I will turn this over to Tammy. 1175 01:00:57.100 --> 01:01:00.010 - Thank you, Sylvia. Hi, everyone. 1176 01:01:00.010 --> 01:01:03.130 Start my video here. 1177 01:01:03.130 --> 01:01:08.130 Not that y'all need to see me, but... (chuckles) 1178 01:01:08.160 --> 01:01:11.785 Welcome. Thank you for joining, as always. 1179 01:01:11.785 --> 01:01:15.620 I'm just going to cover the last few slides here 1180 01:01:15.620 --> 01:01:19.750 and give you a brief sort of general overview 1181 01:01:19.750 --> 01:01:23.560 of our sampling methodology and what we mean 1182 01:01:23.560 --> 01:01:28.420 by period under review and just cover some broad areas 1183 01:01:28.420 --> 01:01:30.460 in those sections. 1184 01:01:30.460 --> 01:01:32.990 So, first off, I just want to remind 1185 01:01:32.990 --> 01:01:35.930 you just our general definitions 1186 01:01:35.930 --> 01:01:37.260 and terminology that we usually use 1187 01:01:37.260 --> 01:01:39.540 when we talk about some of these things. 1188 01:01:39.540 --> 01:01:42.400 So for the sampling frame, as you all probably know, 1189 01:01:42.400 --> 01:01:44.484 we are talking about the whole universe of cases 1190 01:01:44.484 --> 01:01:46.813 from which we draw a random sample, 1191 01:01:47.746 --> 01:01:49.940 and those will include the ones that you pull 1192 01:01:49.940 --> 01:01:50.970 the sample from as well, 1193 01:01:50.970 --> 01:01:54.790 they will comprise the oversample that's used 1194 01:01:54.790 --> 01:01:57.197 later on if you need oversample cases. 1195 01:01:57.197 --> 01:01:58.790 For the sampling period, 1196 01:01:58.790 --> 01:02:00.806 we talk about the time periods in which cases 1197 01:02:00.806 --> 01:02:03.250 from that sample frame are drawn. 1198 01:02:03.250 --> 01:02:05.250 So for the foster care file, 1199 01:02:05.250 --> 01:02:08.170 it will be a 6-month period that begins approximately 1200 01:02:08.170 --> 01:02:11.950 12 months prior to the start of your case review. 1201 01:02:11.950 --> 01:02:14.780 For your in-home services, it's that same month period, 1202 01:02:14.780 --> 01:02:18.103 but we're also adding an additional 45 days. 1203 01:02:18.103 --> 01:02:21.130 This allows cases that happen to be open maybe 1204 01:02:21.130 --> 01:02:23.700 on that last day of the foster care sampling 1205 01:02:23.700 --> 01:02:27.300 period timeframe and open for 45 days, 1206 01:02:27.300 --> 01:02:30.677 it gets allowed to be pulled into possible, 1207 01:02:30.677 --> 01:02:32.800 to the sample to be reviewed. 1208 01:02:32.800 --> 01:02:35.620 The period under review starts at the beginning 1209 01:02:35.620 --> 01:02:37.850 of that sample period and ends when the case 1210 01:02:37.850 --> 01:02:41.310 is reviewed or when the case is closed, 1211 01:02:41.310 --> 01:02:43.403 whichever occurs first. 1212 01:02:46.750 --> 01:02:49.301 And then for the periods under review, 1213 01:02:49.301 --> 01:02:54.260 for Round 4, they were on from 12 to 15 months, 1214 01:02:54.260 --> 01:02:58.670 for CB-led onsite CFSR reviews, 1215 01:02:58.670 --> 01:03:00.870 states will create a foster care sampling frame 1216 01:03:00.870 --> 01:03:01.703 and an in-home service 1217 01:03:01.703 --> 01:03:04.882 sampling frame using a 12-month sampling period 1218 01:03:04.882 --> 01:03:08.170 that starts 12 months prior to the onsite case review. 1219 01:03:08.170 --> 01:03:09.610 I just sort of said that. 1220 01:03:09.610 --> 01:03:13.060 For the state-led, states with review periods longer 1221 01:03:13.060 --> 01:03:15.890 than a month will be expected to renew sampling 1222 01:03:15.890 --> 01:03:17.490 frames monthly or quarterly. 1223 01:03:17.490 --> 01:03:20.630 And I'll talk about that in a second, what that means. 1224 01:03:20.630 --> 01:03:23.700 But as always, CB MASC, we will be available 1225 01:03:23.700 --> 01:03:26.360 to provide consultation assistance and help 1226 01:03:26.360 --> 01:03:29.840 you review plans and validate the sampling frames. 1227 01:03:29.840 --> 01:03:33.720 So you're not on your own for this at all. 1228 01:03:33.720 --> 01:03:36.570 We'll be continuing to do a lot of work with you on that. 1229 01:03:39.360 --> 01:03:43.320 So there's three sampling approaches I'll quickly go over. 1230 01:03:43.320 --> 01:03:47.917 The first one I mentioned earlier is predominantly 1231 01:03:47.917 --> 01:03:49.650 for the CB-led ones, 1232 01:03:49.650 --> 01:03:51.300 although states certainly can do it too, 1233 01:03:51.300 --> 01:03:53.800 there's a fixed sampling approach, 1234 01:03:53.800 --> 01:03:55.980 which really just means you've got one sampling 1235 01:03:55.980 --> 01:03:59.790 period that provides a 12-month PUR for the week 1236 01:03:59.790 --> 01:04:01.960 that you were doing your case reviews, 1237 01:04:01.960 --> 01:04:04.210 and the visual hopefully will help. 1238 01:04:04.210 --> 01:04:07.130 So for example, if you are doing a case review 1239 01:04:07.130 --> 01:04:11.950 that begins in June of 2023, for that week, 1240 01:04:11.950 --> 01:04:14.722 from the 19th to the 23rd, you'll be pulling one 1241 01:04:14.722 --> 01:04:19.380 sample frame that spans that 6-month sampling 1242 01:04:19.380 --> 01:04:22.360 period from June 1st to November 30th. 1243 01:04:22.360 --> 01:04:25.390 From that frame, you will pull your cases, 1244 01:04:25.390 --> 01:04:27.130 and then you will review from the start 1245 01:04:27.130 --> 01:04:30.280 of that sampling frame up until the date of your review, 1246 01:04:30.280 --> 01:04:33.053 which will span a full 12 months. 1247 01:04:34.390 --> 01:04:36.750 Again, just as a reminder for that in-home 1248 01:04:36.750 --> 01:04:39.960 services sampling frame, the extra 45 days will be 1249 01:04:39.960 --> 01:04:41.433 added to that timeframe. 1250 01:04:44.460 --> 01:04:46.350 And I'll try to keep monitoring... 1251 01:04:51.920 --> 01:04:54.180 And so for the rolling monthly sampling frame, 1252 01:04:54.180 --> 01:04:58.380 which I mentioned earlier, as you can see, it is similar, 1253 01:04:58.380 --> 01:05:00.870 many states have done this and are doing it now, 1254 01:05:00.870 --> 01:05:03.270 but the sampling periods advance monthly, 1255 01:05:03.270 --> 01:05:07.100 which provides a full 12-month PUR for each case review, 1256 01:05:07.100 --> 01:05:08.800 and you refresh it every month. 1257 01:05:08.800 --> 01:05:13.800 So it may be a better visual for the next slide. 1258 01:05:17.400 --> 01:05:21.363 So for example, if you are reviewing in April, 1259 01:05:23.012 --> 01:05:25.705 this is really designed for states that do 1260 01:05:25.705 --> 01:05:29.700 case reviews over a number of months with a maximum 1261 01:05:29.700 --> 01:05:31.160 of 6 months. 1262 01:05:31.160 --> 01:05:36.160 So if you start your review in April and you want to roll, 1263 01:05:37.230 --> 01:05:39.530 then your period under review goes from that April 1264 01:05:39.530 --> 01:05:42.280 1st year before to the date of your review. 1265 01:05:42.280 --> 01:05:43.330 And then the next month, 1266 01:05:43.330 --> 01:05:46.270 when you're reviewing in May, your sampling period 1267 01:05:46.270 --> 01:05:48.650 shoots forward a month as well. 1268 01:05:48.650 --> 01:05:51.710 If you just look down the column, 1269 01:05:51.710 --> 01:05:52.970 your sampling periods go 1270 01:05:52.970 --> 01:05:56.920 from April, May, June, July, and they move forward. 1271 01:05:56.920 --> 01:05:59.600 So that's what we call a rolling sampling 1272 01:05:59.600 --> 01:06:00.943 approach for a monthly. 1273 01:06:02.670 --> 01:06:07.070 And then lastly, it's very similar to the monthly, 1274 01:06:07.070 --> 01:06:10.600 only this time you can review quarterly, 1275 01:06:10.600 --> 01:06:12.993 so you roll it quarterly, same concept, 1276 01:06:13.832 --> 01:06:17.060 and then this way it allows, excuse me, 1277 01:06:17.060 --> 01:06:19.960 a PUR for 12 to 15 months, 1278 01:06:19.960 --> 01:06:23.310 so as you move forward quarterly again, 1279 01:06:23.310 --> 01:06:27.815 the table shows that everything kind of shifts forward. 1280 01:06:27.815 --> 01:06:32.815 These rolling approaches often allow states to review, 1281 01:06:33.140 --> 01:06:38.140 to refresh their sample to review more recent case practice. 1282 01:06:38.230 --> 01:06:41.560 So often, a lot of states will opt for that. 1283 01:06:44.690 --> 01:06:49.690 And similar to Round 3, we continue to have 1284 01:06:50.720 --> 01:06:53.283 case elimination criteria. 1285 01:06:54.120 --> 01:06:56.990 There are some minor changes in clarification for Round 4. 1286 01:06:56.990 --> 01:07:00.220 We have a lot of that documentation available 1287 01:07:00.220 --> 01:07:01.053 and it will be available, 1288 01:07:01.053 --> 01:07:05.290 when we start off and all through Round 4. We have some 1289 01:07:05.290 --> 01:07:09.240 CB required and state-specific case elimination criteria. 1290 01:07:09.240 --> 01:07:11.670 Again, through negotiation, much of the time 1291 01:07:11.670 --> 01:07:14.900 between CB and MASC and the Regional Offices, 1292 01:07:14.900 --> 01:07:16.900 we have a process to consistently address 1293 01:07:16.900 --> 01:07:20.700 and obtain approval, and we will expect the process 1294 01:07:20.700 --> 01:07:24.210 to document tracking report case elimination reasoning 1295 01:07:25.530 --> 01:07:27.450 so that everybody can kind of keep track 1296 01:07:27.450 --> 01:07:30.563 of what's happening as you go through that. 1297 01:07:31.780 --> 01:07:33.010 - [Elizabeth] Tammy, we have a few questions 1298 01:07:33.010 --> 01:07:36.580 about the [unintell.] periods and the rolling sampling periods. 1299 01:07:36.580 --> 01:07:39.433 The first one comes from Todd. 1300 01:07:40.310 --> 01:07:42.632 Let me know if you see, if you want me to read it for you. 1301 01:07:42.632 --> 01:07:43.465 - Okay. 1302 01:07:45.190 --> 01:07:50.190 So, Doug, this result in many cases being closed 1303 01:07:50.190 --> 01:07:52.380 by the review time and parents not willing 1304 01:07:52.380 --> 01:07:56.540 to participate and/or not be found. 1305 01:07:56.540 --> 01:07:59.280 I assume that's in relation 1306 01:07:59.280 --> 01:08:02.350 to the rolling sampling timeframes? 1307 01:08:02.350 --> 01:08:05.050 - [Elizabeth] It might be related 1308 01:08:05.050 --> 01:08:06.970 to the fixed sampling period, 1309 01:08:06.970 --> 01:08:10.300 which can be - - Okay. Okay. 1310 01:08:10.300 --> 01:08:11.434 Okay, yeah. 1311 01:08:11.434 --> 01:08:15.070 Again, I think Elizabeth and Sylvia can probably speak 1312 01:08:15.070 --> 01:08:16.100 to this a little bit better, 1313 01:08:16.100 --> 01:08:21.047 but I think there's efforts to be made for, 1314 01:08:22.850 --> 01:08:26.090 to find parents and to review those cases. 1315 01:08:26.090 --> 01:08:29.910 But I believe it's negotiated in terms of case 1316 01:08:29.910 --> 01:08:34.910 elimination criteria on when cases are not being reviewed, 1317 01:08:36.040 --> 01:08:39.450 is that correct, Sylvia and Elizabeth? 1318 01:08:39.450 --> 01:08:42.898 And am I stating that clearly? 1319 01:08:42.898 --> 01:08:45.457 - [Elizabeth] I'm sure with Todd that it is challenging 1320 01:08:45.457 --> 01:08:48.954 having a period under review of 12 to 15 months, 1321 01:08:48.954 --> 01:08:53.210 if cases are closed within a couple months or 4 or 1322 01:08:53.210 --> 01:08:58.063 5 months, it is challenging to locate parents 1323 01:08:58.063 --> 01:09:02.500 and set up the interviews. 1324 01:09:02.500 --> 01:09:04.930 And so that's why we do encourage the in-home 1325 01:09:04.930 --> 01:09:08.610 service sample frame to be at least 4 to 6 times 1326 01:09:08.610 --> 01:09:12.610 the size of the random sample of in-home cases reviewed 1327 01:09:12.610 --> 01:09:14.990 so that you do have a pool of cases. 1328 01:09:14.990 --> 01:09:18.480 And this is similar to Round 3 and also PIP 1329 01:09:18.480 --> 01:09:20.993 measurement that states are involved in right now. 1330 01:09:22.980 --> 01:09:25.100 - Thanks, Elizabeth. I appreciate that. 1331 01:09:25.100 --> 01:09:27.710 If there's a clarifying question you have, 1332 01:09:28.578 --> 01:09:31.219 please just let us know again. 1333 01:09:31.219 --> 01:09:33.700 We had a comment from DC that they have found 1334 01:09:33.700 --> 01:09:36.940 the rolling more useful to capture recent changes 1335 01:09:36.940 --> 01:09:38.270 in case practice. 1336 01:09:38.270 --> 01:09:40.040 We have heard that from a number of states, 1337 01:09:40.040 --> 01:09:42.820 which is why we really are encouraging states 1338 01:09:42.820 --> 01:09:46.563 to do that this time for Round 4, for the onsite reviews. 1339 01:09:47.860 --> 01:09:50.803 I would like to see the previous slide if there's time. 1340 01:09:51.984 --> 01:09:55.470 I'm not sure if that's the previous slide. 1341 01:09:55.470 --> 01:09:56.930 I'm not sure when that question came in. 1342 01:09:56.930 --> 01:10:00.870 I'm sorry, Jennifer, if it's a different slide, 1343 01:10:00.870 --> 01:10:01.870 let me know. 1344 01:10:01.870 --> 01:10:05.240 And with a rolling month procedure, must the number 1345 01:10:05.240 --> 01:10:08.999 of cases reviewed per month be nearly the same? 1346 01:10:08.999 --> 01:10:13.999 We are sort of relaxing that criteria, 1347 01:10:14.950 --> 01:10:17.870 we know for Round 3 that that was the case. 1348 01:10:17.870 --> 01:10:22.870 For Round 4, we are not requiring that, 1349 01:10:25.550 --> 01:10:27.820 although for the CFSR onsite, 1350 01:10:27.820 --> 01:10:30.640 we may want to stick to that a little bit more closely 1351 01:10:30.640 --> 01:10:34.313 than we would for the PIP measurement. 1352 01:10:35.620 --> 01:10:37.970 Elizabeth, do you have anything to add to that? 1353 01:10:39.310 --> 01:10:41.590 - [Elizabeth] No, I think you captured it. 1354 01:10:41.590 --> 01:10:46.590 So it is not going to be the level of scrutiny that we applied in Round 3. 1355 01:10:46.820 --> 01:10:48.880 - Right. Right. 1356 01:10:48.880 --> 01:10:50.920 Okay, Jennifer, I think you said the one before that, 1357 01:10:50.920 --> 01:10:53.343 so hopefully that's the rolling monthly sample. 1358 01:10:55.490 --> 01:10:56.853 What was the question? 1359 01:10:58.460 --> 01:11:01.757 - [Elizabeth] Oh, maybe reading the wrong question. 1360 01:11:01.757 --> 01:11:05.810 - (laughs) I was asked to put up the rolling monthly 1361 01:11:05.810 --> 01:11:07.720 sampling approach slide, I believe. 1362 01:11:07.720 --> 01:11:09.220 That's what someone wanted to see. 1363 01:11:09.220 --> 01:11:14.220 Again, the slides will be available 1364 01:11:14.800 --> 01:11:17.000 with the, getting posted. 1365 01:11:17.000 --> 01:11:18.980 So you'll be able to access them and kind of look 1366 01:11:18.980 --> 01:11:22.460 at these tables to kind of visualize it. 1367 01:11:22.460 --> 01:11:25.840 But again, it's shifting that 6 months forward 1368 01:11:25.840 --> 01:11:28.330 every month, you're dropping off a month and adding 1369 01:11:28.330 --> 01:11:30.430 the following months since you always get rolling 1370 01:11:30.430 --> 01:11:32.173 6-month sampling periods. 1371 01:11:35.531 --> 01:11:36.510 - [Elizabeth] And someone was asking 1372 01:11:36.510 --> 01:11:39.343 for Roger Ward's question that we answered. 1373 01:11:40.380 --> 01:11:41.833 - Okay. Okay. 1374 01:11:42.700 --> 01:11:47.023 You moved the recent one before that. Okay. 1375 01:11:49.280 --> 01:11:50.780 - [Elizabeth] So the question that Roger Ward 1376 01:11:50.780 --> 01:11:54.113 asked Christine was, with a rolling monthly, 1377 01:11:55.225 --> 01:11:58.430 he said procedure or it's the sampling approach, 1378 01:11:58.430 --> 01:12:02.220 must the number of cases reviewed each month be nearly 1379 01:12:02.220 --> 01:12:04.230 the same each month, 1380 01:12:04.230 --> 01:12:06.750 and in Round 3, we did encourage that when states 1381 01:12:06.750 --> 01:12:09.730 rolled monthly that they reviewed a balanced number 1382 01:12:09.730 --> 01:12:12.310 of cases each sampling period. 1383 01:12:12.310 --> 01:12:14.220 And we'll continue to encourage that. 1384 01:12:14.220 --> 01:12:16.740 We're just not going to have the same requirement for scrutiny. 1385 01:12:16.740 --> 01:12:19.199 It can be challenging for scheduling 1386 01:12:19.199 --> 01:12:22.883 to have that requirement. 1387 01:12:26.320 --> 01:12:28.690 - I thought the balance number was eliminated this round. 1388 01:12:28.690 --> 01:12:29.563 That's correct. 1389 01:12:35.560 --> 01:12:38.660 I am not seeing any more questions. 1390 01:12:38.660 --> 01:12:42.960 Hopefully if that slide has been up long enough, I'll just, 1391 01:12:42.960 --> 01:12:45.525 if I need to go back through anything, let me know. 1392 01:12:45.525 --> 01:12:49.860 Again, just some resources as has been mentioned 1393 01:12:49.860 --> 01:12:52.970 by Sylvia and Roger and Jennifer, I believe. 1394 01:12:52.970 --> 01:12:54.580 Technical Bulletin's available. 1395 01:12:54.580 --> 01:12:58.163 We are going to be reissuing some, with it, 1396 01:12:59.460 --> 01:13:02.110 some adjustments to it in the spring. 1397 01:13:02.110 --> 01:13:05.279 We also have the criteria for State-Led Reviews posted. 1398 01:13:05.279 --> 01:13:07.240 Again, that link will be there. 1399 01:13:07.240 --> 01:13:09.910 That's available, I believe, in our CFSR portal, 1400 01:13:09.910 --> 01:13:13.640 as well as the guidance for CFSR site selection proposals. 1401 01:13:13.640 --> 01:13:17.470 And we are continually updating the Procedures Manual 1402 01:13:17.470 --> 01:13:19.363 and that will be coming out soon. 1403 01:13:22.910 --> 01:13:25.033 - [Elizabeth] Okay. A couple more questions. 1404 01:13:27.480 --> 01:13:28.333 - Okay. 1405 01:13:30.734 --> 01:13:33.434 - [Elizabeth] So there's one from Doug, is a statewide 1406 01:13:34.960 --> 01:13:38.853 random sample approach only available for a State-Led Review? 1407 01:13:41.400 --> 01:13:45.730 - Statewide random sample approach only available 1408 01:13:45.730 --> 01:13:47.343 for a State-Led Review. 1409 01:13:49.300 --> 01:13:53.820 That is my understanding that the CB-led onsite 1410 01:13:53.820 --> 01:13:56.673 reviews are the 3 sites for 65 cases. 1411 01:13:57.670 --> 01:14:01.548 That's right. So yes, that is. 1412 01:14:01.548 --> 01:14:02.590 - Jennifer- - Somebody wants 1413 01:14:02.590 --> 01:14:04.643 the definition slide. - Okay. 1414 01:14:06.520 --> 01:14:10.993 - I believe that might be what they're looking for. 1415 01:14:14.560 --> 01:14:15.910 Slightly confused by the difference 1416 01:14:15.910 --> 01:14:19.083 between the quarterly rolling and the monthly rolling. 1417 01:14:22.890 --> 01:14:25.540 Similar concept here, you're absolutely right. 1418 01:14:25.540 --> 01:14:27.580 So for the rolling monthly, 1419 01:14:27.580 --> 01:14:29.540 you would refresh your sample every month, 1420 01:14:29.540 --> 01:14:33.170 going back to the month that you're now doing. 1421 01:14:33.170 --> 01:14:34.003 Right? 1422 01:14:34.003 --> 01:14:37.180 So I apologize, whoever wanted to see the definitions, 1423 01:14:37.180 --> 01:14:40.490 I'm going to just really quickly scroll through. 1424 01:14:40.490 --> 01:14:42.250 So for the rolling monthly, for example, 1425 01:14:42.250 --> 01:14:45.320 your 6-month sampling period will run 1426 01:14:45.320 --> 01:14:49.344 from April to September, but then you would go again 1427 01:14:49.344 --> 01:14:52.250 in May and your sampling period 1428 01:14:52.250 --> 01:14:54.840 then shifts from May to October. 1429 01:14:54.840 --> 01:14:58.060 So everything kind of shifts forward, as you can see, 1430 01:14:58.060 --> 01:15:02.020 from that graphic, depending on what month 1431 01:15:02.020 --> 01:15:03.710 that you're doing your case review, right? 1432 01:15:03.710 --> 01:15:04.940 Goes up to 6 months. 1433 01:15:04.940 --> 01:15:08.930 You're always getting 1 year behind you, 1434 01:15:08.930 --> 01:15:11.750 but it's rolling every month. 1435 01:15:11.750 --> 01:15:13.500 And so for quarterly, however, 1436 01:15:13.500 --> 01:15:15.010 it's the same kind of concept, 1437 01:15:15.010 --> 01:15:18.240 but the 6-month period spans 2 quarters. 1438 01:15:18.240 --> 01:15:21.440 So you don't advance every month between quarter 1 1439 01:15:21.440 --> 01:15:26.410 and quarter 2, you're advancing every 3 months. 1440 01:15:26.410 --> 01:15:29.570 So your first period would run, for example, 1441 01:15:29.570 --> 01:15:33.870 from April to September. 1442 01:15:33.870 --> 01:15:36.910 And then you would shift forward those 3 months 1443 01:15:36.910 --> 01:15:39.123 from July to December. 1444 01:15:40.020 --> 01:15:44.083 And you would review those 2 quarters and look back. 1445 01:15:44.940 --> 01:15:47.150 Hopefully that's a little... 1446 01:15:49.600 --> 01:15:51.480 "Repeat the last question and answer 1447 01:15:51.480 --> 01:15:53.520 about the State-Led Review." 1448 01:15:53.520 --> 01:15:57.279 The question about the State-Led Review was, 1449 01:15:57.279 --> 01:16:02.279 is the option to pull a statewide random sample 1450 01:16:03.110 --> 01:16:08.110 for the onsite CFSR review only available to state-led? 1451 01:16:08.257 --> 01:16:11.380 And the answer to that question is yes. 1452 01:16:11.380 --> 01:16:14.370 For the CB-led, which we previously talked 1453 01:16:14.370 --> 01:16:16.610 about federal onsite reviews, 1454 01:16:16.610 --> 01:16:21.326 the CB-Led Reviews are limited to the 65 cases 1455 01:16:21.326 --> 01:16:24.090 and the 3 sites that are identified 1456 01:16:24.090 --> 01:16:26.713 for your onsite CFSR review. 1457 01:16:28.910 --> 01:16:30.990 Oh, and I'm glad you got the, 1458 01:16:30.990 --> 01:16:32.830 I was glad I was somewhat clear 1459 01:16:32.830 --> 01:16:34.560 on the quarterly rolling sample. 1460 01:16:34.560 --> 01:16:37.470 The rolling samples can be confusing, and again, 1461 01:16:37.470 --> 01:16:40.370 MASC is here to guide you through and help you as best 1462 01:16:40.370 --> 01:16:41.300 we can on this. 1463 01:16:41.300 --> 01:16:44.620 We recognize it's cumbersome and we'll definitely 1464 01:16:44.620 --> 01:16:46.070 be working very closely in the states 1465 01:16:46.070 --> 01:16:48.530 on how to identify their sample frames, 1466 01:16:48.530 --> 01:16:51.267 how to move things forward, and how to pull 1467 01:16:51.267 --> 01:16:54.670 your samples and verification of the cases that should 1468 01:16:54.670 --> 01:16:56.223 be in those sampling frames. 1469 01:17:01.420 --> 01:17:04.850 I believe the last slide, there's a question slide. 1470 01:17:04.850 --> 01:17:09.850 So any others on these, looks like there are a couple. 1471 01:17:09.950 --> 01:17:11.693 Can't hear you. I'm sorry. 1472 01:17:16.310 --> 01:17:20.150 "Will rolling samples be required for State-Led Reviews 1473 01:17:20.150 --> 01:17:24.463 or will fixed samples still be allowed?" 1474 01:17:28.810 --> 01:17:32.880 We don't require, and correct me if I'm wrong, someone, 1475 01:17:32.880 --> 01:17:37.880 we don't require that states roll, we encourage it. 1476 01:17:38.570 --> 01:17:41.343 We strongly encourage it. Go ahead, Elizabeth. 1477 01:17:42.750 --> 01:17:46.310 - [Elizabeth] If, Karen, you have a review schedule 1478 01:17:46.310 --> 01:17:51.310 that spans beyond 3 months of the 6-month 1479 01:17:51.510 --> 01:17:54.040 maximum that's allowed for a State-Led Review, 1480 01:17:54.040 --> 01:17:58.260 then we do want you to use a rolling sampling approach, 1481 01:17:58.260 --> 01:18:01.180 but there are some State-Led Reviews that do a review 1482 01:18:01.180 --> 01:18:02.855 in a concentrated period of time, 1483 01:18:02.855 --> 01:18:05.471 just 2 months or 3 months or 1, 1484 01:18:05.471 --> 01:18:08.490 there's a few states that just do it in a couple weeks, 1485 01:18:08.490 --> 01:18:11.883 and in that case, they could use a fixed sample. 1486 01:18:25.006 --> 01:18:26.530 - Have the phones been placed on mute? 1487 01:18:26.530 --> 01:18:28.581 I can't hear any more. 1488 01:18:28.581 --> 01:18:30.840 I hope folks can hear me, Christine. 1489 01:18:30.840 --> 01:18:35.840 It might just be, I think people are hearing, I'm not sure. 1490 01:18:38.140 --> 01:18:41.769 Roger, when we run the data profile syntax. 1491 01:18:41.769 --> 01:18:44.080 Sometimes there are very slight differences 1492 01:18:44.080 --> 01:18:49.080 between our results and the results supplied by our data team. 1493 01:18:49.470 --> 01:18:54.160 "Can you provide your SPSS files on our data to us 1494 01:18:54.160 --> 01:18:56.210 so that we can compare, 1495 01:18:56.210 --> 01:18:58.810 kind of a loss in explaining the slight differences." 1496 01:19:00.427 --> 01:19:03.910 I'm going to leave that answer 1497 01:19:03.910 --> 01:19:07.270 to Jennifer and Roger, because we don't 1498 01:19:07.270 --> 01:19:12.270 traditionally need to return state files to them, 1499 01:19:13.730 --> 01:19:18.730 but I will let the higher- ups answer that one. (laughs) 1500 01:19:23.660 --> 01:19:25.520 - [Elizabeth] "Can we provide your SPSS files 1501 01:19:25.520 --> 01:19:26.763 on our data to us?" 1502 01:19:29.370 --> 01:19:34.370 Roger, we hear this question and understand the concern. 1503 01:19:36.880 --> 01:19:37.880 And I think 1504 01:19:40.410 --> 01:19:42.640 that we have resources, including the Center 1505 01:19:42.640 --> 01:19:45.403 for States, who will also have the data files. 1506 01:19:46.570 --> 01:19:51.240 So if you have persistent differences that you can't explain, 1507 01:19:51.240 --> 01:19:52.930 then I think we can, 1508 01:19:52.930 --> 01:19:55.590 you should reach out to the CB data team versus - 1509 01:19:56.950 --> 01:20:01.000 Well, start with the CBC for States, 1510 01:20:01.000 --> 01:20:05.120 but also you could check in with the CB data team 1511 01:20:05.120 --> 01:20:07.290 through their mailbox and we can try and see 1512 01:20:07.290 --> 01:20:10.493 if there's something that explains the difference. 1513 01:20:13.650 --> 01:20:14.530 At the very least, 1514 01:20:14.530 --> 01:20:18.130 when you say that - you mean the SPSS output, I imagine, 1515 01:20:18.130 --> 01:20:19.490 and I think that's something that we have had 1516 01:20:19.490 --> 01:20:20.700 on our radar for some time. 1517 01:20:20.700 --> 01:20:23.383 So I think we could figure out how to help you with that. 1518 01:20:24.380 --> 01:20:27.570 But I'd like for the data team to correct me 1519 01:20:27.570 --> 01:20:30.523 if I'm saying something that is not so. 1520 01:20:32.800 --> 01:20:35.110 - [Roger] That sounds great. I think, yeah. 1521 01:20:35.110 --> 01:20:38.503 Initially work the Capacity Building Center. 1522 01:20:43.210 --> 01:20:44.110 If they're not able to help you, 1523 01:20:44.110 --> 01:20:49.110 then we can try to help find where the difference is. 1524 01:20:49.658 --> 01:20:52.520 - Yeah. - But yeah, 1525 01:20:52.520 --> 01:20:54.357 they have, the Capacity Building Center has the files. 1526 01:20:54.357 --> 01:20:56.880 I don't think we have a way to send the data actually 1527 01:20:56.880 --> 01:21:00.340 back out. I don't know if you want to do that. 1528 01:21:05.645 --> 01:21:06.478 - Great. 1529 01:21:08.690 --> 01:21:13.494 Jennifer, I believe Elizabeth answered the difference 1530 01:21:13.494 --> 01:21:15.840 between the rolling and the fixed, 1531 01:21:15.840 --> 01:21:19.863 which kind of leads us to the CB-led versus state-led. 1532 01:21:21.029 --> 01:21:23.390 I'm not sure if you have an actual question 1533 01:21:23.390 --> 01:21:26.410 about that or you just needed to see the slide again, 1534 01:21:26.410 --> 01:21:29.963 but if you have a particular question, just let us know. 1535 01:21:31.330 --> 01:21:32.163 Yeah. 1536 01:21:32.163 --> 01:21:35.620 I recognize this sound was choppy and I think 1537 01:21:35.620 --> 01:21:39.760 it's tough when we have almost 400 people 1538 01:21:39.760 --> 01:21:43.950 on and I think there's just some computer issues. 1539 01:21:43.950 --> 01:21:46.920 We are going to work our best to resolve that. 1540 01:21:46.920 --> 01:21:50.080 But I agree, the phone seems to be a better way 1541 01:21:50.080 --> 01:21:51.853 to have heard us. 1542 01:21:53.140 --> 01:21:56.663 I really hope CB starts using Zoom instead of Adobe Connect. 1543 01:21:57.920 --> 01:21:58.753 Yeah. 1544 01:21:58.753 --> 01:22:01.963 We've heard that as well. Thank you for that. 1545 01:22:05.390 --> 01:22:07.540 - Tammy, did you see Jennifer's comment 1546 01:22:07.540 --> 01:22:09.640 about the state-led period under review difference 1547 01:22:09.640 --> 01:22:11.400 from CB is what she was looking at? 1548 01:22:11.400 --> 01:22:12.233 Oh, you did. 1549 01:22:12.233 --> 01:22:14.110 - I did. I believe this is slide 22. 1550 01:22:14.110 --> 01:22:18.440 I believe Elizabeth was answering that, whether... 1551 01:22:18.440 --> 01:22:22.950 I think it's the fixed versus the rolling 1552 01:22:22.950 --> 01:22:27.713 or that the CB-led is 12 months, right, from the... 1553 01:22:29.700 --> 01:22:32.893 And then it goes to 12 to 15 if you're rolling. 1554 01:22:37.630 --> 01:22:38.723 I'm not sure. 1555 01:22:40.440 --> 01:22:42.820 - Jennifer. - All right. Oh, I'm sorry. 1556 01:22:42.820 --> 01:22:44.410 There's a comment. There's a follow-up. 1557 01:22:44.410 --> 01:22:47.693 I was just confused by the 12 months and 1-month review, 1558 01:22:48.560 --> 01:22:51.083 for state-led it is the same type of PUR. 1559 01:22:54.830 --> 01:22:57.350 - [Elizabeth] For CB-led, Jennifer, all the states 1560 01:22:57.350 --> 01:23:01.583 will have 12-month periods under review, but if states are, 1561 01:23:02.750 --> 01:23:04.400 set up their case review schedule, 1562 01:23:04.400 --> 01:23:06.270 or it spans more than 1 month, 1563 01:23:06.270 --> 01:23:11.270 or a lot of times review cases over 3 to 6 months, 1564 01:23:11.630 --> 01:23:14.510 and then we would want you to renew your sample 1565 01:23:14.510 --> 01:23:17.550 so you don't end up with periods under review longer 1566 01:23:17.550 --> 01:23:18.803 than 15 months. 1567 01:23:25.210 --> 01:23:30.210 - Right. Because the onsite, we're onsite for 1 week. 1568 01:23:31.055 --> 01:23:32.923 So it's just that 12 months. 1569 01:23:35.980 --> 01:23:38.580 Hopefully I believe that... 1570 01:23:39.630 --> 01:23:42.003 Looks like we've got 3 minutes. 1571 01:23:43.610 --> 01:23:46.080 I don't see any other questions rolling in. 1572 01:23:46.080 --> 01:23:49.750 Does anybody have any closing comments? 1573 01:23:49.750 --> 01:23:52.520 Thank you for your time and your patience from my part. 1574 01:23:52.520 --> 01:23:56.706 I appreciate it. I'm going to pop off video. 1575 01:23:56.706 --> 01:23:57.890 Oh. 1576 01:23:57.890 --> 01:24:00.070 - [Jennifer] Okay. You can leave the video, Tammy. 1577 01:24:00.070 --> 01:24:04.120 I just wanted to thank everyone, again, for coming. 1578 01:24:04.120 --> 01:24:07.500 We will pay close attention to all of your questions 1579 01:24:07.500 --> 01:24:10.530 and answers, to make sure we answered the questions 1580 01:24:10.530 --> 01:24:12.740 about the content, and we'll address 1581 01:24:12.740 --> 01:24:17.260 your concerns about the sound. 1582 01:24:17.260 --> 01:24:21.332 I did just want to note that we will be posting not only 1583 01:24:21.332 --> 01:24:26.100 the transcript - or not the transcript, 1584 01:24:26.100 --> 01:24:28.010 the audio of this meeting, 1585 01:24:28.010 --> 01:24:30.370 as well as the PowerPoint so that you have access 1586 01:24:30.370 --> 01:24:31.850 to these resources. 1587 01:24:31.850 --> 01:24:34.410 And then, of course, please stay tuned. 1588 01:24:34.410 --> 01:24:37.220 We'll be meeting with you again in the coming weeks, 1589 01:24:37.220 --> 01:24:41.700 but as we move toward spring and the spring months, 1590 01:24:41.700 --> 01:24:45.992 we will be issuing more resources, as the team mentioned, 1591 01:24:45.992 --> 01:24:49.593 both the Procedures Manual and an updated TB #13. 1592 01:24:50.570 --> 01:24:52.320 So I think that's it for me. 1593 01:24:52.320 --> 01:24:56.970 I do want to thank you all for your attention, 1594 01:24:56.970 --> 01:25:01.920 for your patience, and thank my team and the JBS team 1595 01:25:01.920 --> 01:25:05.470 for their efforts too, as well, with this presentation. 1596 01:25:05.470 --> 01:25:09.220 So thank you. And I think - - Thanks Jen. 1597 01:25:09.220 --> 01:25:11.490 - Unless there are more questions. - I just, 1598 01:25:11.490 --> 01:25:13.110 Elizabeth or somebody just put up 1599 01:25:13.110 --> 01:25:14.510 the last slide, my apologies. 1600 01:25:14.510 --> 01:25:17.710 I forgot just to remind everybody that we do have 1601 01:25:17.710 --> 01:25:21.450 the Part 2, today was focused on the onsite CFSR 1602 01:25:21.450 --> 01:25:23.460 review process, but the Part 2 will focus 1603 01:25:23.460 --> 01:25:25.780 on PIP measurement and sampling, 1604 01:25:25.780 --> 01:25:28.043 and we will talk about the framework, 1605 01:25:29.210 --> 01:25:31.060 the statewide data indicators, and the case 1606 01:25:31.060 --> 01:25:34.320 review populations, and how you set baselines, goals, 1607 01:25:34.320 --> 01:25:35.700 and goals achievement. 1608 01:25:35.700 --> 01:25:37.650 We'll talk a little bit about Item 1, 1609 01:25:38.520 --> 01:25:41.230 the option to use an aggregate measure and some 1610 01:25:41.230 --> 01:25:43.310 timeframes, and have some time for discussions 1611 01:25:43.310 --> 01:25:45.610 and questions, but just to remind you all 1612 01:25:45.610 --> 01:25:48.697 that we have that call, next week on March 9th. 1613 01:25:55.721 --> 01:25:57.137 I guess, thank you.