Compare commits

..

51 Commits

Author SHA1 Message Date
2bd7e3b970 Updated login page.
All checks were successful
ci / build-test (push) Successful in 2m29s
ci / publish-image (push) Successful in 1m44s
2026-05-04 01:52:35 -04:00
abcc82437f Updated logic for getting released work information (take 60 day period max limit into consideration). 2026-05-04 01:52:25 -04:00
f6674e0382 Added "Delete Voice Works" functionality. 2026-05-04 01:51:40 -04:00
77a02a543d Added tag filter state persistence.
All checks were successful
ci / build-test (push) Successful in 2m26s
ci / publish-image (push) Successful in 1m32s
2026-04-26 23:39:47 -04:00
2355d7fe65 Added front-end "Set English Tag Name" logic.
All checks were successful
ci / build-test (push) Successful in 2m27s
ci / publish-image (push) Successful in 1m27s
2026-04-26 00:40:30 -04:00
204e186354 Added tag/creator status update functionality on the API and UI layers.
All checks were successful
ci / build-test (push) Successful in 2m29s
ci / publish-image (push) Successful in 1m42s
2026-04-25 14:17:13 -04:00
dbed9fc905 Updated packages.
All checks were successful
ci / build-test (push) Successful in 2m32s
ci / publish-image (push) Successful in 1m32s
2026-04-25 10:19:44 -04:00
d6a4015c91 Added logging to scanner.
All checks were successful
ci / build-test (push) Successful in 2m29s
ci / publish-image (push) Successful in 1m32s
2026-04-24 10:18:26 -04:00
b63a89c8be Added new pagination component. 2026-04-24 10:18:16 -04:00
dfa840d816 Updated packages.
All checks were successful
ci / build-test (push) Successful in 2m35s
ci / publish-image (push) Successful in 2m44s
2026-04-21 20:12:25 -04:00
6bc91b293d Updated tags/creators page styling.
All checks were successful
ci / build-test (push) Successful in 2m30s
ci / publish-image (push) Successful in 1m42s
2026-04-21 00:41:31 -04:00
da33973229 Updated tags and creators pages.
All checks were successful
ci / build-test (push) Successful in 2m30s
ci / publish-image (push) Successful in 1m46s
2026-04-20 00:36:53 -04:00
c203b2cbdb Updated Blazor UI tag/creator views. Altered logic for sorting favorite/blacklisted fields for tags/creators.
All checks were successful
ci / build-test (push) Successful in 2m45s
ci / publish-image (push) Successful in 1m58s
2026-04-18 21:39:58 -04:00
1f91e46527 Improved performance of voice work search updater.
All checks were successful
ci / build-test (push) Successful in 2m56s
ci / publish-image (push) Successful in 2m15s
2026-04-17 10:29:23 -04:00
be466b52e0 Updated packages.
All checks were successful
ci / build-test (push) Successful in 4m45s
ci / publish-image (push) Successful in 3m0s
2026-04-15 09:55:57 -04:00
b4863a9edf Updated packages.
Some checks failed
ci / build-test (push) Failing after 3m34s
ci / publish-image (push) Has been skipped
2026-04-14 20:52:05 -04:00
b13340061f Fixed duplicate tag/creator upsert issue.
Some checks failed
ci / build-test (push) Failing after 13m47s
ci / publish-image (push) Has been cancelled
2026-04-13 20:15:13 -04:00
85a28a6017 Updates various packages.
All checks were successful
ci / build-test (push) Successful in 2m25s
ci / publish-image (push) Successful in 3m5s
2026-04-12 12:06:01 -04:00
45a8c8be5a Added voice work repository tests.
All checks were successful
ci / build-test (push) Successful in 2m21s
ci / publish-image (push) Successful in 1m39s
2026-03-31 00:43:02 -04:00
347f6f297d If the ingest comes in with zero supported language, do not make any changes, specifically removals.
All checks were successful
ci / build-test (push) Successful in 2m23s
ci / publish-image (push) Successful in 1m37s
2026-03-30 23:22:50 -04:00
adfbf654a6 Added logic to remove supported languages that are no longer supported, rather than just being purely additive. Added ApiClient logging.
All checks were successful
ci / build-test (push) Successful in 2m30s
ci / publish-image (push) Successful in 2m1s
2026-03-30 23:03:53 -04:00
0dd11e6351 Added English localization logic during the regular scan process.
All checks were successful
ci / build-test (push) Successful in 2m17s
ci / publish-image (push) Successful in 1m41s
2026-03-29 21:24:04 -04:00
d9e421178f Added inital job entity and services. Added released works API integration.
All checks were successful
ci / build-test (push) Successful in 2m21s
ci / publish-image (push) Successful in 2m19s
2026-03-27 01:32:39 -04:00
1c016ac62e Undid favorite/blacklisted tag color-coded stylings.
All checks were successful
ci / build-test (push) Successful in 2m44s
ci / publish-image (push) Successful in 1m32s
2026-03-18 00:27:43 -04:00
ce9fbe491d Fixed "Filter Language" test.
All checks were successful
ci / build-test (push) Successful in 2m19s
ci / publish-image (push) Successful in 1m49s
2026-03-17 00:31:08 -04:00
22d5a261c5 Fixed voice work supported language search. Include supported languages and original circle in the search result item. Several UI style updates.
Some checks failed
ci / build-test (push) Failing after 2m52s
ci / publish-image (push) Has been skipped
2026-03-17 00:07:02 -04:00
c8403e0e21 Updated various packages.
All checks were successful
ci / build-test (push) Successful in 2m18s
ci / publish-image (push) Successful in 2m2s
2026-03-14 22:42:53 -04:00
a45f08fe6d Merge pull request 'Chobit Integration' (#2) from feature/issue-1-chobit-integration into master
All checks were successful
ci / build-test (push) Successful in 2m34s
ci / publish-image (push) Successful in 2m49s
Reviewed-on: #2
2026-03-15 02:34:11 +00:00
928e69b2ec Updated launch settings for debugging, and removed redundant project reference.
All checks were successful
ci / build-test (pull_request) Successful in 2m35s
ci / publish-image (pull_request) Has been skipped
2026-03-14 22:24:21 -04:00
aab7bee694 Added Chobit integration. Updated tests.
All checks were successful
ci / build-test (push) Successful in 2m27s
ci / publish-image (push) Has been skipped
2026-03-14 21:46:53 -04:00
ee809e374f Updated packages.
All checks were successful
ci / build-test (push) Successful in 2m29s
ci / publish-image (push) Successful in 2m38s
2026-03-10 20:54:27 -04:00
ddd0738cb5 Fixed formatting issues.
All checks were successful
ci / build-test (push) Successful in 2m43s
ci / publish-image (push) Successful in 3m49s
2026-03-09 01:18:17 -04:00
1cbadd3042 Updated CI to publish-images on push to default branch. 2026-03-09 01:13:52 -04:00
a8dd1fa6aa Updated CI workflow.
All checks were successful
ci / build-test (push) Successful in 2m59s
ci / publish-image (push) Has been skipped
2026-03-09 01:02:20 -04:00
62c2efab01 Updated scanner to infer when it has reached the end of results.
All checks were successful
ci / build-test (push) Successful in 2m18s
ci / publish-image (push) Has been skipped
2026-03-07 01:26:04 -05:00
1e01edf1b7 Updated worker to show upsert issue messages.
All checks were successful
ci / build-test (push) Successful in 2m36s
ci / publish-image (push) Has been skipped
2026-03-07 00:03:18 -05:00
1d40013837 Fixed voice work updater bug. Added integration tests for voice work search updates (Japanese).
All checks were successful
ci / build-test (push) Successful in 2m27s
ci / publish-image (push) Has been skipped
2026-03-05 23:29:29 -05:00
61f2e64972 Updated scanner and table names.
All checks were successful
ci / build-test (push) Successful in 2m38s
ci / publish-image (push) Has been skipped
2026-03-05 20:56:57 -05:00
79bece9e1c Updated appsettings for the worker app. Added split querying for updaters.
All checks were successful
ci / build-test (push) Successful in 2m27s
ci / publish-image (push) Has been skipped
2026-03-01 22:48:41 -05:00
83655f13e9 Updated various parts of scanning and ingestion, either for bug fixes, or for enhancements.
All checks were successful
ci / build-test (push) Successful in 2m22s
ci / publish-image (push) Has been skipped
2026-03-01 22:07:20 -05:00
704a6fc433 Updated scanner logic to handle thumb VueJS components. Removed uneeded DLSiteWork fields.
All checks were successful
ci / build-test (push) Successful in 2m21s
ci / publish-image (push) Has been skipped
2026-02-28 22:20:24 -05:00
ca7ffa1730 Updated CI to use .NET 10 SDK. 2026-02-25 23:18:56 -05:00
ae8d7d34d9 Migrated from .NET 9 to .NET 10.
Some checks failed
ci / publish-image (push) Has been skipped
ci / build-test (push) Failing after 1m32s
2026-02-25 20:18:38 -05:00
0bfbc17a43 Updated various packages.
All checks were successful
ci / build-test (push) Successful in 2m20s
ci / publish-image (push) Has been skipped
2026-02-24 22:29:31 -05:00
ab3524ea20 Added docker-compose. Updated startups for API and Web layer.
Some checks failed
ci / build-test (push) Has been cancelled
ci / publish-image (push) Has been cancelled
2026-02-24 00:25:03 -05:00
80ca1296e5 Updated front-end authentication. 2026-02-22 21:47:34 -05:00
8348603b13 Added voice work image fallback. Added tag/creator/circle chip components. Updated voice work search response to include favorite/blacklisted flags for tags/creators/circles.
Some checks failed
ci / build-test (push) Has been cancelled
ci / publish-image (push) Has been cancelled
2026-02-22 01:56:04 -05:00
9f30ef446a Added authenication/authorization. Refactored api startup.
Some checks failed
ci / build-test (push) Has been cancelled
ci / publish-image (push) Has been cancelled
2026-02-16 00:20:02 -05:00
a85989a337 Fixed scanning issue. Updated worker.
Some checks failed
ci / build-test (push) Has been cancelled
ci / publish-image (push) Has been cancelled
2026-02-14 22:47:19 -05:00
340c62d18b Added worker app.
All checks were successful
ci / build-test (push) Successful in 2m16s
ci / publish-image (push) Has been skipped
2026-02-01 21:41:23 -05:00
c51775592e Added subtitle language logic.
All checks were successful
ci / build-test (push) Successful in 2m15s
ci / publish-image (push) Has been skipped
2025-12-22 23:33:46 -05:00
209 changed files with 12036 additions and 842 deletions

View File

@@ -7,14 +7,18 @@ on:
- 'JSMR.Application/**' - 'JSMR.Application/**'
- 'JSMR.Domain/**' - 'JSMR.Domain/**'
- 'JSMR.Infrastructure/**' - 'JSMR.Infrastructure/**'
- 'JSMR.UI.Blazor/**'
- 'JSMR.Tests/**' - 'JSMR.Tests/**'
- '.gitea/workflows/ci.yml'
pull_request: pull_request:
paths: paths:
- 'JSMR.Api/**' - 'JSMR.Api/**'
- 'JSMR.Application/**' - 'JSMR.Application/**'
- 'JSMR.Domain/**' - 'JSMR.Domain/**'
- 'JSMR.Infrastructure/**' - 'JSMR.Infrastructure/**'
- 'JSMR.UI.Blazor/**'
- 'JSMR.Tests/**' - 'JSMR.Tests/**'
- '.gitea/workflows/ci.yml'
jobs: jobs:
build-test: build-test:
@@ -31,7 +35,7 @@ jobs:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4 - uses: actions/setup-dotnet@v4
with: with:
dotnet-version: '9.0.x' dotnet-version: '10.0.x'
- name: Docker sanity (ensures socket mount is working) - name: Docker sanity (ensures socket mount is working)
run: docker version run: docker version
@@ -44,7 +48,8 @@ jobs:
- run: dotnet test --configuration Release --no-build --logger "trx;LogFileName=test-results.trx" - run: dotnet test --configuration Release --no-build --logger "trx;LogFileName=test-results.trx"
publish-image: publish-image:
if: ${{ false }} # disabled for now if: github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
needs: build-test
runs-on: [self-hosted, linux, x64, docker] runs-on: [self-hosted, linux, x64, docker]
container: container:
image: ghcr.io/catthehacker/ubuntu:act-latest image: ghcr.io/catthehacker/ubuntu:act-latest
@@ -52,19 +57,20 @@ jobs:
--privileged --privileged
env: env:
REGISTRY: ${{ secrets.REGISTRY_HOST }} REGISTRY: ${{ secrets.REGISTRY_HOST }}
OWNER_REPO: ${{ github.repository }}
DOCKERFILE: JSMR.Api/Dockerfile
CONTEXT: .
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
- name: Normalize image name (lowercase) - name: Set image variables
id: names id: vars
shell: bash
run: | run: |
IMAGE_LC="$(echo "${REGISTRY}/${OWNER_REPO}" | tr '[:upper:]' '[:lower:]')" OWNER_LC="$(echo "${{ github.repository_owner }}" | tr '[:upper:]' '[:lower:]')"
echo "image=${IMAGE_LC}" >> "$GITHUB_OUTPUT" SHA_SHORT="$(echo "${{ github.sha }}" | cut -c1-7)"
echo "owner_lc=$OWNER_LC" >> "$GITHUB_OUTPUT"
echo "sha_short=$SHA_SHORT" >> "$GITHUB_OUTPUT"
- name: Docker login - name: Docker login
shell: bash
run: | run: |
echo "${{ secrets.REGISTRY_PASSWORD }}" \ echo "${{ secrets.REGISTRY_PASSWORD }}" \
| docker login "${{ env.REGISTRY }}" \ | docker login "${{ env.REGISTRY }}" \
@@ -74,11 +80,32 @@ jobs:
- name: Enable BuildKit - name: Enable BuildKit
run: echo "DOCKER_BUILDKIT=1" >> $GITHUB_ENV run: echo "DOCKER_BUILDKIT=1" >> $GITHUB_ENV
- name: Build - name: Build API image
run: docker build -f "$DOCKERFILE" -t "${{ steps.names.outputs.image }}:${{ github.sha }}" "$CONTEXT" shell: bash
- name: Push
run: | run: |
docker push "${{ steps.names.outputs.image }}:${{ github.sha }}" docker build \
docker tag "${{ steps.names.outputs.image }}:${{ github.sha }}" "${{ steps.names.outputs.image }}:latest" -f JSMR.Api/Dockerfile \
docker push "${{ steps.names.outputs.image }}:latest" -t "${{ env.REGISTRY }}/${{ steps.vars.outputs.owner_lc }}/jsmr-api:${{ steps.vars.outputs.sha_short }}" \
-t "${{ env.REGISTRY }}/${{ steps.vars.outputs.owner_lc }}/jsmr-api:main" \
.
- name: Push API image
shell: bash
run: |
docker push "${{ env.REGISTRY }}/${{ steps.vars.outputs.owner_lc }}/jsmr-api:${{ steps.vars.outputs.sha_short }}"
docker push "${{ env.REGISTRY }}/${{ steps.vars.outputs.owner_lc }}/jsmr-api:main"
- name: Build Web image
shell: bash
run: |
docker build \
-f JSMR.UI.Blazor/Dockerfile \
-t "${{ env.REGISTRY }}/${{ steps.vars.outputs.owner_lc }}/jsmr-web:${{ steps.vars.outputs.sha_short }}" \
-t "${{ env.REGISTRY }}/${{ steps.vars.outputs.owner_lc }}/jsmr-web:main" \
.
- name: Push Web image
shell: bash
run: |
docker push "${{ env.REGISTRY }}/${{ steps.vars.outputs.owner_lc }}/jsmr-web:${{ steps.vars.outputs.sha_short }}"
docker push "${{ env.REGISTRY }}/${{ steps.vars.outputs.owner_lc }}/jsmr-web:main"

3
.gitignore vendored
View File

@@ -361,3 +361,6 @@ MigrationBackup/
# Fody - auto-generated XML schema # Fody - auto-generated XML schema
FodyWeavers.xsd FodyWeavers.xsd
# Environment
.env

View File

@@ -1,5 +1,5 @@
# --- Build stage --- # --- Build stage ---
FROM mcr.microsoft.com/dotnet/sdk:9.0 AS build FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src WORKDIR /src
# copy solution + project files first (for caching) # copy solution + project files first (for caching)
@@ -16,7 +16,7 @@ COPY . .
RUN dotnet publish JSMR.Api/JSMR.Api.csproj -c Release -o /app/publish --no-restore RUN dotnet publish JSMR.Api/JSMR.Api.csproj -c Release -o /app/publish --no-restore
# --- Runtime stage --- # --- Runtime stage ---
FROM mcr.microsoft.com/dotnet/aspnet:9.0 AS runtime FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime
WORKDIR /app WORKDIR /app
COPY --from=build /app/publish ./ COPY --from=build /app/publish ./
ENV ASPNETCORE_URLS=http://+:8080 ENV ASPNETCORE_URLS=http://+:8080

View File

@@ -1,17 +1,17 @@
<Project Sdk="Microsoft.NET.Sdk.Web"> <Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup> <PropertyGroup>
<TargetFramework>net9.0</TargetFramework> <TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable> <Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings> <ImplicitUsings>enable</ImplicitUsings>
<UserSecretsId>35cebc06-af6a-44cf-aa71-ecdaf1edc82b</UserSecretsId> <UserSecretsId>35cebc06-af6a-44cf-aa71-ecdaf1edc82b</UserSecretsId>
</PropertyGroup> </PropertyGroup>
<ItemGroup> <ItemGroup>
<PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="9.0.10" /> <PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="10.0.7" />
<PackageReference Include="Serilog" Version="4.3.0" /> <PackageReference Include="Serilog" Version="4.3.1" />
<PackageReference Include="Serilog.AspNetCore" Version="9.0.0" /> <PackageReference Include="Serilog.AspNetCore" Version="10.0.0" />
<PackageReference Include="Serilog.Settings.Configuration" Version="9.0.0" /> <PackageReference Include="Serilog.Settings.Configuration" Version="10.0.0" />
<PackageReference Include="Serilog.Sinks.Seq" Version="9.0.0" /> <PackageReference Include="Serilog.Sinks.Seq" Version="9.0.0" />
</ItemGroup> </ItemGroup>

View File

@@ -1,6 +1,31 @@
@host = http://localhost:5226 @host = http://localhost:5226
@contentType = application/json @contentType = application/json
### Login
POST {{host}}/auth/login
Content-Type: {{contentType}}
{
"Username": "brister",
"Password": "password"
}
### Logout
POST {{host}}/auth/logout
Content-Type: {{contentType}}
{
}
### Get current user
GET {{host}}/api/me
Content-Type: {{contentType}}
{
}
### Search tags by name ### Search tags by name
POST {{host}}/api/tags/search POST {{host}}/api/tags/search
Content-Type: {{contentType}} Content-Type: {{contentType}}

View File

@@ -1,158 +1,26 @@
using JSMR.Application.Circles.Queries.Search; using JSMR.Api.Startup;
using JSMR.Application.Creators.Queries.Search;
using JSMR.Application.DI;
using JSMR.Application.Tags.Queries.Search;
using JSMR.Application.VoiceWorks.Queries.Search;
using JSMR.Infrastructure.Data;
using JSMR.Infrastructure.DI;
using Microsoft.AspNetCore.Http.Json;
using Microsoft.EntityFrameworkCore;
using Serilog;
using Serilog.Events;
using System.Diagnostics;
using System.Text.Json;
using System.Text.Json.Serialization;
var builder = WebApplication.CreateBuilder(args); WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
ConfigurationManager configuration = builder.Configuration;
IWebHostEnvironment environment = builder.Environment;
// Add services to the container.
builder.Services builder.Services
.AddMemoryCache() // TODO .AddAppServices(configuration)
.AddApplication() .AddAppJson()
.AddInfrastructure(); .AddAppOpenApi()
.AddAppAuthentication(environment)
.AddAppCors(configuration);
//.AddAppLogging(builder);
// DbContext (MySQL here; swap to Npgsql when you migrate) builder.Host.UseAppSerilog();
var cs = builder.Configuration.GetConnectionString("AppDb")
?? throw new InvalidOperationException("Missing ConnectionStrings:AppDb2");
builder.Services.AddDbContextFactory<AppDbContext>(opt => WebApplication app = builder.Build();
opt.UseMySql(cs, ServerVersion.AutoDetect(cs)) app.UseAppPipeline(app.Environment);
.EnableSensitiveDataLogging(false));
builder.Services.AddControllers();
// Learn more about configuring OpenAPI at https://aka.ms/aspnet/openapi
builder.Services.AddOpenApi();
builder.Services.Configure<JsonOptions>(options =>
{
options.SerializerOptions.PropertyNameCaseInsensitive = true;
options.SerializerOptions.Converters.Add(
new JsonStringEnumConverter(JsonNamingPolicy.CamelCase)); // or null for exact names
});
// Serilog bootstrap (before Build)
Log.Logger = new LoggerConfiguration()
.ReadFrom.Configuration(builder.Configuration)
.MinimumLevel.Override("Microsoft", LogEventLevel.Warning)
.Enrich.WithProperty("Service", "JSMR.Api")
.Enrich.WithProperty("Environment", builder.Environment.EnvironmentName)
.CreateLogger();
builder.Host.UseSerilog();
builder.Services.AddCors(o =>
{
o.AddPolicy("ui-dev", p => p
.AllowAnyOrigin()
//.WithOrigins(
// "*",
// "https://localhost:5173", // vite-like
// "https://localhost:5001", // typical https dev
// "http://localhost:5000", // typical http dev
// "https://localhost:7112", // blazor wasm dev https (adjust to your port)
// "http://localhost:5153" // blazor wasm dev http (adjust to your port)
//)
.AllowAnyHeader()
.AllowAnyMethod());
});
var app = builder.Build();
app.UseCors("ui-dev");
// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment()) if (app.Environment.IsDevelopment())
{ await app.SeedDevelopmentAsync();
app.MapOpenApi();
}
app.UseHttpsRedirection(); app.MapAppEndpoints();
app.UseAuthorization();
app.MapControllers();
// Request logging with latency, status, path
app.UseSerilogRequestLogging(opts =>
{
opts.MessageTemplate = "HTTP {RequestMethod} {RequestPath} responded {StatusCode} in {Elapsed:0.0000} ms";
});
// Correlation: ensure every log has traceId and return it to clients
app.Use(async (ctx, next) =>
{
// Use current Activity if present (W3C trace context), else fall back
var traceId = Activity.Current?.TraceId.ToString() ?? ctx.TraceIdentifier;
using (Serilog.Context.LogContext.PushProperty("TraceId", traceId))
{
ctx.Response.Headers["x-trace-id"] = traceId;
await next();
}
});
// Health check
app.MapGet("/health", () => Results.Ok(new { status = "ok" }));
// ---- Endpoints ----
// Circles Search
app.MapPost("/api/circles/search", async (
SearchCirclesRequest request,
SearchCirclesHandler handler,
CancellationToken cancallationToken) =>
{
try
{
SearchCirclesResponse result = await handler.HandleAsync(request, cancallationToken);
return Results.Ok(result);
}
catch (OperationCanceledException) when (cancallationToken.IsCancellationRequested)
{
return Results.StatusCode(StatusCodes.Status499ClientClosedRequest);
}
});
// Voice Works Search
app.MapPost("/api/voiceworks/search", async (
SearchVoiceWorksRequest request,
SearchVoiceWorksHandler handler,
CancellationToken cancallationToken) =>
{
var result = await handler.HandleAsync(request, cancallationToken);
return Results.Ok(result);
});
// Tags Search
app.MapPost("/api/tags/search", async (
SearchTagsRequest request,
SearchTagsHandler handler,
CancellationToken cancallationToken) =>
{
var result = await handler.HandleAsync(request, cancallationToken);
return Results.Ok(result);
});
// Creators Search
app.MapPost("/api/creators/search", async (
SearchCreatorsRequest request,
SearchCreatorsHandler handler,
CancellationToken cancallationToken) =>
{
var result = await handler.HandleAsync(request, cancallationToken);
return Results.Ok(result);
});
app.Run(); app.Run();

View File

@@ -0,0 +1,30 @@
using JSMR.Domain.Entities;
using JSMR.Infrastructure.Data;
using Microsoft.EntityFrameworkCore;
namespace JSMR.Api.Startup;
public static class DevSeeding
{
public static async Task SeedDevelopmentAsync(this WebApplication app)
{
using var scope = app.Services.CreateScope();
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
var username = "brister";
if (!await db.Users.AnyAsync(u => u.Username == username))
{
db.Users.Add(new User
{
Username = username,
PasswordHash = BCrypt.Net.BCrypt.HashPassword("password"),
Role = "Admin",
IsActive = true
});
await db.SaveChangesAsync();
Console.WriteLine("✅ Seeded development user: brister / password");
}
}
}

View File

@@ -0,0 +1,31 @@
using Serilog;
using Serilog.Events;
namespace JSMR.Api.Startup;
public static class HostBuilderExtensions
{
public static IHostBuilder UseAppSerilog(this IHostBuilder host)
{
return host.UseSerilog((context, services, loggerConfiguration) =>
{
IConfiguration configuration = context.Configuration;
IHostEnvironment environment = context.HostingEnvironment;
loggerConfiguration
.ReadFrom.Configuration(configuration)
.ReadFrom.Services(services)
.MinimumLevel.Override("Microsoft", LogEventLevel.Warning)
.Enrich.WithProperty("Service", "JSMR.Api")
.Enrich.WithProperty("Environment", environment.EnvironmentName);
// Conditionally add Seq if configured correctly
string? seqUrl = configuration["Seq:ServerUrl"];
if (Uri.TryCreate(seqUrl, UriKind.Absolute, out _))
{
loggerConfiguration.WriteTo.Seq(seqUrl);
}
});
}
}

View File

@@ -0,0 +1,130 @@
using JSMR.Application.DI;
using JSMR.Infrastructure.Data;
using JSMR.Infrastructure.DI;
using Microsoft.AspNetCore.Authentication.Cookies;
using Microsoft.AspNetCore.Http.Json;
using Microsoft.EntityFrameworkCore;
using System.Text.Json;
using System.Text.Json.Serialization;
namespace JSMR.Api.Startup;
public static class ServiceCollectionExtensions
{
public static IServiceCollection AddAppServices(this IServiceCollection services, IConfigurationManager configuration)
{
services
.AddMemoryCache()
.AddApplication()
.AddInfrastructure();
string connectionString = configuration.GetConnectionString("AppDb")
?? throw new InvalidOperationException("Missing ConnectionStrings:AppDb");
services.AddDbContextFactory<AppDbContext>(opt =>
opt.UseMySql(connectionString, ServerVersion.AutoDetect(connectionString))
.EnableSensitiveDataLogging(false));
services.AddControllers();
return services;
}
public static IServiceCollection AddAppJson(this IServiceCollection services)
{
services.Configure<JsonOptions>(options =>
{
options.SerializerOptions.PropertyNameCaseInsensitive = true;
options.SerializerOptions.Converters.Add(
new JsonStringEnumConverter(JsonNamingPolicy.CamelCase));
});
return services;
}
public static IServiceCollection AddAppOpenApi(this IServiceCollection services)
{
services.AddOpenApi();
return services;
}
public static IServiceCollection AddAppAuthentication(this IServiceCollection services, IHostEnvironment environment)
{
services.AddAuthorization();
services
.AddAuthentication(CookieAuthenticationDefaults.AuthenticationScheme)
.AddCookie(options =>
{
options.Cookie.Name = "vw_auth";
options.Cookie.HttpOnly = true;
//options.Cookie.SameSite = SameSiteMode.None;
//options.Cookie.SecurePolicy = CookieSecurePolicy.Always;
options.Cookie.SameSite = SameSiteMode.Lax;
options.Cookie.SecurePolicy =
environment.IsDevelopment()
? CookieSecurePolicy.SameAsRequest
: CookieSecurePolicy.Always;
options.Events = new CookieAuthenticationEvents
{
OnRedirectToLogin = ctx =>
{
ctx.Response.StatusCode = StatusCodes.Status401Unauthorized;
return Task.CompletedTask;
},
OnRedirectToAccessDenied = ctx =>
{
ctx.Response.StatusCode = StatusCodes.Status403Forbidden;
return Task.CompletedTask;
}
};
});
return services;
}
public static IServiceCollection AddAppCors(this IServiceCollection services, IConfigurationManager configuration)
{
string[] origins = configuration.GetSection("Cors:AllowedOrigins").Get<string[]>() ?? [];
services.AddCors(options =>
{
options.AddPolicy("ui", policyBuilder =>
{
if (origins.Length == 0)
{
// In container/prod you often don't need CORS at all (same-origin),
// but if it *is* needed and not configured, fail closed rather than crash.
// Do not call WithOrigins().
return;
}
policyBuilder.WithOrigins(origins)
.AllowAnyHeader()
.AllowAnyMethod()
.AllowCredentials();
});
});
return services;
}
//public static IServiceCollection AddAppLogging(this IServiceCollection services, IHostApplicationBuilder builder)
//{
// var config = builder.Configuration;
// var env = builder.Environment;
// Log.Logger = new LoggerConfiguration()
// .ReadFrom.Configuration(config)
// .MinimumLevel.Override("Microsoft", LogEventLevel.Warning)
// .Enrich.WithProperty("Service", "JSMR.Api")
// .Enrich.WithProperty("Environment", env.EnvironmentName)
// .CreateLogger();
// return services;
//}
}

View File

@@ -0,0 +1,189 @@
using JSMR.Application.Circles.Queries.Search;
using JSMR.Application.Creators.Commands.UpdateCreatorStatus;
using JSMR.Application.Creators.Queries.Search;
using JSMR.Application.Tags.Commands.SetEnglishName;
using JSMR.Application.Tags.Commands.UpdateTagStatus;
using JSMR.Application.Tags.Queries.Search;
using JSMR.Application.Users;
using JSMR.Application.VoiceWorks.Queries.Search;
using Microsoft.AspNetCore.Authentication;
using Microsoft.AspNetCore.Authentication.Cookies;
using Serilog;
using System.Diagnostics;
using System.Security.Claims;
namespace JSMR.Api.Startup;
public static class WebApplicationExtensions
{
public static WebApplication UseAppPipeline(this WebApplication app, IHostEnvironment env)
{
string[] origins = app.Configuration.GetSection("Cors:AllowedOrigins").Get<string[]>() ?? [];
if (origins.Length > 0)
app.UseCors("ui");
if (env.IsDevelopment())
app.MapOpenApi();
if (!env.IsDevelopment())
{
app.UseHttpsRedirection();
}
app.UseAuthentication();
app.UseAuthorization();
app.MapControllers();
app.UseSerilogRequestLogging(opts =>
{
opts.MessageTemplate = "HTTP {RequestMethod} {RequestPath} responded {StatusCode} in {Elapsed:0.0000} ms";
});
app.Use(async (ctx, next) =>
{
var traceId = Activity.Current?.TraceId.ToString() ?? ctx.TraceIdentifier;
using (Serilog.Context.LogContext.PushProperty("TraceId", traceId))
{
ctx.Response.Headers["x-trace-id"] = traceId;
await next();
}
});
return app;
}
public static void MapAppEndpoints(this WebApplication app)
{
app.MapGet("/health", () => Results.Ok(new { status = "ok" }));
app.MapSearchEndpoints();
app.MapTagCommandEndpoints();
app.MapCreatorCommandEndpoints();
app.MapAuthenticationEndpoints();
}
private static void MapSearchEndpoints(this WebApplication app)
{
app.MapPost("/api/circles/search", async (
SearchCirclesRequest request,
SearchCirclesHandler handler,
CancellationToken ct) =>
{
try
{
var result = await handler.HandleAsync(request, ct);
return Results.Ok(result);
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
return Results.StatusCode(StatusCodes.Status499ClientClosedRequest);
}
});
app.MapPost("/api/voiceworks/search", async (
SearchVoiceWorksRequest request,
SearchVoiceWorksHandler handler,
CancellationToken ct) =>
{
var result = await handler.HandleAsync(request, ct);
return Results.Ok(result);
});
app.MapPost("/api/tags/search", async (
SearchTagsRequest request,
SearchTagsHandler handler,
CancellationToken ct) =>
{
var result = await handler.HandleAsync(request, ct);
return Results.Ok(result);
});
app.MapPost("/api/creators/search", async (
SearchCreatorsRequest request,
SearchCreatorsHandler handler,
CancellationToken ct) =>
{
var result = await handler.HandleAsync(request, ct);
return Results.Ok(result);
});
}
private static void MapTagCommandEndpoints(this WebApplication app)
{
app.MapPost("/api/tags/update-status", async (
UpdateTagStatusRequest request,
UpdateTagStatusHandler handler,
CancellationToken ct) =>
{
var result = await handler.HandleAsync(request, ct);
return Results.Ok(result);
});
app.MapPost("/api/tags/set-english-name", async (
SetTagEnglishNameRequest request,
SetTagEnglishNameHandler handler,
CancellationToken ct) =>
{
var result = await handler.HandleAsync(request, ct);
return Results.Ok(result);
});
}
private static void MapCreatorCommandEndpoints(this WebApplication app)
{
app.MapPost("/api/creators/update-status", async (
UpdateCreatorStatusRequest request,
UpdateCreatorStatusHandler handler,
CancellationToken ct) =>
{
var result = await handler.HandleAsync(request, ct);
return Results.Ok(result);
});
}
private static void MapAuthenticationEndpoints(this WebApplication app)
{
app.MapPost("/auth/login", async (LoginRequest req, IUserRepository users, HttpContext http) =>
{
var user = await users.FindByUsernameAsync(req.Username);
if (user is null || !user.IsActive)
return Results.Unauthorized();
if (!users.VerifyPassword(user, req.Password))
return Results.Unauthorized();
var claims = new List<Claim>
{
new(ClaimTypes.NameIdentifier, user.Id.ToString()),
new(ClaimTypes.Name, user.Username),
new(ClaimTypes.Role, user.Role ?? "User")
};
var identity = new ClaimsIdentity(claims, CookieAuthenticationDefaults.AuthenticationScheme);
var principal = new ClaimsPrincipal(identity);
await http.SignInAsync(CookieAuthenticationDefaults.AuthenticationScheme, principal);
return Results.Ok(new { ok = true });
});
app.MapPost("/auth/logout", async (HttpContext http) =>
{
await http.SignOutAsync(CookieAuthenticationDefaults.AuthenticationScheme);
return Results.Ok(new { ok = true });
});
app.MapGet("/api/me", (ClaimsPrincipal user) =>
{
return Results.Ok(new
{
name = user.Identity?.Name,
id = user.FindFirstValue(ClaimTypes.NameIdentifier),
role = user.FindFirstValue(ClaimTypes.Role)
});
}).RequireAuthorization();
}
public record LoginRequest(string Username, string Password);
}

View File

@@ -6,6 +6,15 @@
} }
}, },
"AllowedHosts": "*", "AllowedHosts": "*",
"Cors": {
"AllowedOrigins": [
"https://localhost:5173", // vite-like
"https://localhost:5001", // typical https dev
"http://localhost:5000", // typical http dev
"https://localhost:7112", // blazor wasm dev https (adjust to your port)
"http://localhost:5153" // blazor wasm dev http (adjust to your port)
]
},
"Serilog": { "Serilog": {
"MinimumLevel": { "MinimumLevel": {
"Default": "Information", "Default": "Information",
@@ -16,11 +25,10 @@
}, },
"Enrich": [ "FromLogContext", "WithMachineName", "WithProcessId", "WithThreadId" ], "Enrich": [ "FromLogContext", "WithMachineName", "WithProcessId", "WithThreadId" ],
"WriteTo": [ "WriteTo": [
{ "Name": "Console" }, { "Name": "Console" }
{
"Name": "Seq",
"Args": { "serverUrl": "%SEQ_URL%" }
}
] ]
},
"Seq": {
"ServerUrl": ""
} }
} }

View File

@@ -2,9 +2,9 @@
public record CreatorSearchItem public record CreatorSearchItem
{ {
public int CreatorId { get; init; } public int CreatorId { get; set; }
public required string Name { get; init; } public required string Name { get; set; }
public bool Favorite { get; init; } public bool Favorite { get; set; }
public bool Blacklisted { get; init; } public bool Blacklisted { get; set; }
public int VoiceWorkCount { get; init; } public int VoiceWorkCount { get; set; }
} }

View File

@@ -1,5 +1,7 @@
using JSMR.Application.Circles.Queries.Search; using JSMR.Application.Circles.Queries.Search;
using JSMR.Application.Creators.Commands.UpdateCreatorStatus;
using JSMR.Application.Creators.Queries.Search; using JSMR.Application.Creators.Queries.Search;
using JSMR.Application.Scanning;
using JSMR.Application.Tags.Commands.SetEnglishName; using JSMR.Application.Tags.Commands.SetEnglishName;
using JSMR.Application.Tags.Commands.UpdateTagStatus; using JSMR.Application.Tags.Commands.UpdateTagStatus;
using JSMR.Application.Tags.Queries.Search; using JSMR.Application.Tags.Queries.Search;
@@ -16,13 +18,14 @@ public static class ApplicationServiceCollectionExtensions
services.AddScoped<SearchCirclesHandler>(); services.AddScoped<SearchCirclesHandler>();
services.AddScoped<SearchVoiceWorksHandler>(); services.AddScoped<SearchVoiceWorksHandler>();
//services.AddScoped<ScanVoiceWorksHandler>(); services.AddScoped<ScanVoiceWorksHandler>();
services.AddScoped<SearchTagsHandler>(); services.AddScoped<SearchTagsHandler>();
services.AddScoped<SetTagEnglishNameHandler>(); services.AddScoped<SetTagEnglishNameHandler>();
services.AddScoped<UpdateTagStatusHandler>(); services.AddScoped<UpdateTagStatusHandler>();
services.AddScoped<SearchCreatorsHandler>(); services.AddScoped<SearchCreatorsHandler>();
services.AddScoped<UpdateCreatorStatusHandler>();
return services; return services;
} }

View File

@@ -1,7 +1,12 @@
namespace JSMR.Application.Integrations.Chobit.Models; using System.Text.Json.Serialization;
namespace JSMR.Application.Integrations.Chobit.Models;
public class ChobitResult public class ChobitResult
{ {
[JsonPropertyName("count")]
public int Count { get; set; } public int Count { get; set; }
[JsonPropertyName("works")]
public ChobitWork[] Works { get; set; } = []; public ChobitWork[] Works { get; set; } = [];
} }

View File

@@ -0,0 +1,3 @@
namespace JSMR.Application.Integrations.Chobit.Models;
public class ChobitResultCollection : Dictionary<string, ChobitResult> { }

View File

@@ -1,16 +1,39 @@
namespace JSMR.Application.Integrations.Chobit.Models; using System.Text.Json.Serialization;
namespace JSMR.Application.Integrations.Chobit.Models;
public class ChobitWork public class ChobitWork
{ {
[JsonPropertyName("work_id")]
public string? WorkId { get; set; } public string? WorkId { get; set; }
[JsonPropertyName("dlsite_work_id")]
public string? DLSiteWorkId { get; set; } public string? DLSiteWorkId { get; set; }
[JsonPropertyName("work_name")]
public string? WorkName { get; set; } public string? WorkName { get; set; }
[JsonPropertyName("work_name_kana")]
public string? WorkNameKana { get; set; } public string? WorkNameKana { get; set; }
public string? URL { get; set; }
public string? EmbedURL { get; set; } [JsonPropertyName("url")]
public string? Url { get; set; }
[JsonPropertyName("embed_url")]
public string? EmbedUrl { get; set; }
[JsonPropertyName("thumb")]
public string? Thumb { get; set; } public string? Thumb { get; set; }
[JsonPropertyName("mini_thumb")]
public string? MiniThumb { get; set; } public string? MiniThumb { get; set; }
[JsonPropertyName("file_type")]
public string? FileType { get; set; } public string? FileType { get; set; }
[JsonPropertyName("embed_width")]
public decimal EmbedWidth { get; set; } public decimal EmbedWidth { get; set; }
[JsonPropertyName("embed_height")]
public decimal EmbedHeight { get; set; } public decimal EmbedHeight { get; set; }
} }

View File

@@ -1,3 +0,0 @@
namespace JSMR.Application.Integrations.Chobit.Models;
public class ChobitWorkResult : Dictionary<string, ChobitResult> { }

View File

@@ -0,0 +1,9 @@
using JSMR.Application.Integrations.Chobit.Models;
namespace JSMR.Application.Integrations.Chobit.Ports;
public interface IChobitClient
{
Task<ChobitResult> GetSampleInfoAsync(string productId, CancellationToken cancellationToken = default);
Task<ChobitResultCollection> GetSampleInfoAsync(string[] productIds, CancellationToken cancellationToken = default);
}

View File

@@ -1,6 +1,6 @@
using JSMR.Application.Integrations.Cien.Models; using JSMR.Application.Integrations.Cien.Models;
namespace JSMR.Application.Integrations.Ports; namespace JSMR.Application.Integrations.Cien.Ports;
public interface ICienClient public interface ICienClient
{ {

View File

@@ -0,0 +1,10 @@
namespace JSMR.Application.Integrations.DLSite.Models.ReleasedWorks;
public record ReleasedWork
{
public required string ProductId { get; init; }
public required string Title { get; init; }
public required string MaskedTitle { get; init; }
public required string Description { get; init; }
public required string MaskedDescription { get; init; }
}

View File

@@ -0,0 +1,9 @@
using JSMR.Application.Enums;
namespace JSMR.Application.Integrations.DLSite.Models.ReleasedWorks;
public record ReleasedWorksRequest(
Locale Locale,
DateOnly Date,
int Period
);

View File

@@ -0,0 +1,6 @@
namespace JSMR.Application.Integrations.DLSite.Models.ReleasedWorks;
public class ReleasedWorksCollection : Dictionary<string, ReleasedWork>
{
}

View File

@@ -5,6 +5,7 @@ namespace JSMR.Application.Integrations.DLSite.Models;
public class VoiceWorkDetails public class VoiceWorkDetails
{ {
public string? Title { get; init; }
public VoiceWorkSeries? Series { get; init; } public VoiceWorkSeries? Series { get; init; }
public VoiceWorkTranslation? Translation { get; init; } public VoiceWorkTranslation? Translation { get; init; }
public AgeRating AgeRating { get; init; } public AgeRating AgeRating { get; init; }

View File

@@ -0,0 +1,10 @@
using JSMR.Application.Integrations.DLSite.Models;
using JSMR.Application.Integrations.DLSite.Models.ReleasedWorks;
namespace JSMR.Application.Integrations.DLSite.Ports;
public interface IDLSiteClient
{
Task<VoiceWorkDetailCollection> GetVoiceWorkDetailsAsync(string[] productIds, CancellationToken cancellationToken = default);
Task<ReleasedWorksCollection> GetReleasedWorksAsync(ReleasedWorksRequest request, CancellationToken cancellationToken);
}

View File

@@ -1,8 +0,0 @@
using JSMR.Application.Integrations.Chobit.Models;
namespace JSMR.Application.Integrations.Ports;
public interface IChobitClient
{
Task<ChobitWorkResult> GetSampleInfoAsync(string[] productIds, CancellationToken cancellationToken = default);
}

View File

@@ -1,8 +0,0 @@
using JSMR.Application.Integrations.DLSite.Models;
namespace JSMR.Application.Integrations.Ports;
public interface IDLSiteClient
{
Task<VoiceWorkDetailCollection> GetVoiceWorkDetailsAsync(string[] productIds, CancellationToken cancellationToken = default);
}

View File

@@ -1,7 +1,7 @@
<Project Sdk="Microsoft.NET.Sdk"> <Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup> <PropertyGroup>
<TargetFramework>net9.0</TargetFramework> <TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings> <ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable> <Nullable>enable</Nullable>
</PropertyGroup> </PropertyGroup>
@@ -11,8 +11,8 @@
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="9.0.10" /> <PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.7" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="9.0.10" /> <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.7" />
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>

View File

@@ -0,0 +1,10 @@
namespace JSMR.Application.Jobs;
public interface IJobProgressWriter
{
Task SetStepAsync(int jobId, string step, CancellationToken cancellationToken);
Task SetProgressAsync(int jobId, int? current, int? total, CancellationToken cancellationToken);
Task SetHeartbeatAsync(int jobId, CancellationToken cancellationToken);
Task CompleteAsync(int jobId, string? summary, CancellationToken cancellationToken);
Task FailAsync(int jobId, string error, CancellationToken cancellationTokenct);
}

View File

@@ -0,0 +1,15 @@
using JSMR.Domain.Entities;
namespace JSMR.Application.Jobs;
public interface IJobRepository
{
Task<Job> AddAsync(Job job, CancellationToken cancellationToken);
Task<Job?> GetByIdAsync(int id, CancellationToken cancellationToken);
Task<IReadOnlyList<Job>> GetRecentAsync(int take, CancellationToken cancellationToken);
Task<bool> AnyRunningAsync(CancellationToken cancellationToken);
Task<Job?> TryClaimNextQueuedAsync(string workerName, CancellationToken cancellationToken);
Task SaveChangesAsync(CancellationToken cancellationToken);
}

View File

@@ -4,11 +4,8 @@ namespace JSMR.Application.Scanning.Contracts;
public class DLSiteWork public class DLSiteWork
{ {
public DLSiteWorkType Type { get; set; }
public DLSiteWorkCategory Category { get; set; }
public required string ProductName { get; set; } public required string ProductName { get; set; }
public required string ProductId { get; set; } public required string ProductId { get; set; }
public DateOnly? AnnouncedDate { get; set; }
public DateOnly? ExpectedDate { get; set; } public DateOnly? ExpectedDate { get; set; }
public DateOnly? SalesDate { get; set; } public DateOnly? SalesDate { get; set; }
public int Downloads { get; set; } public int Downloads { get; set; }

View File

@@ -1,7 +1,7 @@
namespace JSMR.Application.Scanning.Contracts; namespace JSMR.Application.Scanning.Contracts;
public enum DLSiteWorkType //public enum DLSiteWorkType
{ //{
Released, // Released,
Announced // Announced
} //}

View File

@@ -1,4 +1,5 @@
using JSMR.Application.Integrations.DLSite.Models; using JSMR.Application.Integrations.Chobit.Models;
using JSMR.Application.Integrations.DLSite.Models;
using JSMR.Domain.Enums; using JSMR.Domain.Enums;
using JSMR.Domain.ValueObjects; using JSMR.Domain.ValueObjects;
@@ -16,7 +17,7 @@ public sealed record VoiceWorkIngest
public int WishlistCount { get; init; } public int WishlistCount { get; init; }
public int Downloads { get; init; } public int Downloads { get; init; }
public bool HasTrial { get; init; } public bool HasTrial { get; init; }
public bool HasDLPlay { get; init; } public bool HasChobit { get; init; }
public byte? StarRating { get; init; } public byte? StarRating { get; init; }
public int? Votes { get; init; } public int? Votes { get; init; }
public AgeRating AgeRating { get; init; } public AgeRating AgeRating { get; init; }
@@ -28,22 +29,23 @@ public sealed record VoiceWorkIngest
public AIGeneration AI { get; init; } public AIGeneration AI { get; init; }
public VoiceWorkSeries? Series { get; init; } public VoiceWorkSeries? Series { get; init; }
public VoiceWorkTranslation? Translation { get; init; } public VoiceWorkTranslation? Translation { get; init; }
public VoiceWorkLocalizationIngest[] Localizations { get; init; } = [];
public static VoiceWorkIngest From(DLSiteWork work, VoiceWorkDetails? details) public static VoiceWorkIngest From(DLSiteWork work, VoiceWorkDetails? details, ChobitResult? chobit)
{ {
return new VoiceWorkIngest() return new VoiceWorkIngest()
{ {
MakerId = work.MakerId, MakerId = work.MakerId,
MakerName = work.Maker, MakerName = work.Maker,
ProductId = work.ProductId, ProductId = work.ProductId,
Title = work.ProductName, Title = details?.Title ?? work.ProductName,
Description = work.Description ?? string.Empty, Description = work.Description ?? string.Empty,
Tags = work.Tags, Tags = work.Tags,
Creators = work.Creators, Creators = work.Creators,
WishlistCount = details?.WishlistCount ?? 0, WishlistCount = details?.WishlistCount ?? 0,
Downloads = Math.Max(work.Downloads, details?.DownloadCount ?? 0), Downloads = Math.Max(work.Downloads, details?.DownloadCount ?? 0),
HasTrial = work.HasTrial || (details?.HasTrial ?? false), HasTrial = work.HasTrial || (details?.HasTrial ?? false),
HasDLPlay = details?.HasDLPlay ?? false, HasChobit = chobit?.Count > 0,
StarRating = work.StarRating, StarRating = work.StarRating,
Votes = work.Votes, Votes = work.Votes,
AgeRating = details?.AgeRating ?? work.AgeRating, AgeRating = details?.AgeRating ?? work.AgeRating,

View File

@@ -0,0 +1,10 @@
using JSMR.Domain.Enums;
namespace JSMR.Application.Scanning.Contracts;
public sealed class VoiceWorkLocalizationIngest
{
public Language Language { get; init; }
public string? Title { get; init; }
public string? Description { get; init; }
}

View File

@@ -0,0 +1,6 @@
namespace JSMR.Application.Scanning.Contracts;
public record VoiceWorkScanResult(
DLSiteWork[] Works,
bool EndOfResults
);

View File

@@ -0,0 +1,9 @@
using JSMR.Application.Integrations.DLSite.Models.ReleasedWorks;
using JSMR.Application.Scanning.Contracts;
namespace JSMR.Application.Scanning.Ports;
public interface IReleasedWorksProvider
{
Task<ReleasedWorksCollection> GetReleasedWorksAsync(VoiceWorkScanResult scanResult, CancellationToken cancellationToken);
}

View File

@@ -0,0 +1,8 @@
using JSMR.Application.Scanning.Contracts;
namespace JSMR.Application.Scanning.Ports;
public interface IVoiceWorkIngestBuilder
{
Task<VoiceWorkIngest[]> BuildAsync(VoiceWorkScanResult scanResult, CancellationToken cancellationToken);
}

View File

@@ -0,0 +1,7 @@
using JSMR.Application.Enums;
namespace JSMR.Application.Scanning.Ports;
public interface IVoiceWorkScannerRepository
{
public IVoiceWorksScanner? GetScanner(Locale locale);
}

View File

@@ -1,4 +1,5 @@
using JSMR.Application.Scanning.Contracts; using JSMR.Application.Scanning.Contracts;
using JSMR.Domain.Enums;
namespace JSMR.Application.Scanning.Ports; namespace JSMR.Application.Scanning.Ports;
@@ -9,7 +10,10 @@ public interface IVoiceWorkUpdater
public class VoiceWorkUpsertResult public class VoiceWorkUpsertResult
{ {
public string ProductId { get; set; } = string.Empty;
public string Title { get; set; } = string.Empty;
public int? VoiceWorkId { get; set; } public int? VoiceWorkId { get; set; }
public VoiceWorkStatus UpdateStatus { get; set; }
public ICollection<VoiceWorkUpsertIssue> Issues { get; } = []; public ICollection<VoiceWorkUpsertIssue> Issues { get; } = [];
public VoiceWorkUpsertStatus Status { get; set; } = VoiceWorkUpsertStatus.Unchanged; public VoiceWorkUpsertStatus Status { get; set; } = VoiceWorkUpsertStatus.Unchanged;
} }

View File

@@ -0,0 +1,8 @@
using JSMR.Application.Enums;
namespace JSMR.Application.Scanning.Ports;
public interface IVoiceWorkUpdaterRepository
{
public IVoiceWorkUpdater? GetUpdater(Locale locale);
}

View File

@@ -4,5 +4,5 @@ namespace JSMR.Application.Scanning.Ports;
public interface IVoiceWorksScanner public interface IVoiceWorksScanner
{ {
Task<IReadOnlyList<DLSiteWork>> ScanPageAsync(VoiceWorkScanOptions request, CancellationToken cancellationToken = default); Task<VoiceWorkScanResult> ScanPageAsync(VoiceWorkScanOptions request, CancellationToken cancellationToken = default);
} }

View File

@@ -1,53 +1,163 @@
using JSMR.Application.Common.Caching; using JSMR.Application.Common.Caching;
using JSMR.Application.Integrations.DLSite.Models;
using JSMR.Application.Integrations.Ports;
using JSMR.Application.Scanning.Contracts; using JSMR.Application.Scanning.Contracts;
using JSMR.Application.Scanning.Ports; using JSMR.Application.Scanning.Ports;
using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging;
using System.Diagnostics;
namespace JSMR.Application.Scanning; namespace JSMR.Application.Scanning;
public sealed class ScanVoiceWorksHandler( public sealed class ScanVoiceWorksHandler(
IServiceProvider serviceProvider, ILogger<ScanVoiceWorksHandler> logger,
IDLSiteClient dlsiteClient, IVoiceWorkScannerRepository scannerRepository,
IVoiceWorkUpdaterRepository updaterRepository,
ISpamCircleCache spamCircleCache, ISpamCircleCache spamCircleCache,
IVoiceWorkIngestBuilder ingestBuilder,
IVoiceWorkSearchUpdater searchUpdater) IVoiceWorkSearchUpdater searchUpdater)
{ {
public async Task<ScanVoiceWorksResponse> HandleAsync(ScanVoiceWorksRequest request, CancellationToken cancellationToken) public async Task<ScanVoiceWorksResponse> HandleAsync(ScanVoiceWorksRequest request, CancellationToken cancellationToken)
{ {
IVoiceWorksScanner? scanner = serviceProvider.GetKeyedService<IVoiceWorksScanner>(request.Locale); using IDisposable? scope = logger.BeginScope(new Dictionary<string, object>
IVoiceWorkUpdater? updater = serviceProvider.GetKeyedService<IVoiceWorkUpdater>(request.Locale); {
["Locale"] = request.Locale,
["PageNumber"] = request.PageNumber,
["PageSize"] = request.PageSize
});
if (scanner is null || updater is null) Stopwatch stopwatch = Stopwatch.StartNew();
return new(); string currentPhase = "initialization";
try
{
logger.LogInformation(
"Starting scan handler for page {PageNumber}, page size {PageSize}, locale {Locale}",
request.PageNumber,
request.PageSize,
request.Locale);
currentPhase = "resolve_scanner";
IVoiceWorksScanner? scanner = scannerRepository.GetScanner(request.Locale);
currentPhase = "resolve_updater";
IVoiceWorkUpdater? updater = updaterRepository.GetUpdater(request.Locale);
if (scanner is null)
throw new InvalidOperationException($"No scanner registered for locale {request.Locale}.");
if (updater is null)
throw new InvalidOperationException($"No updater registered for locale {request.Locale}.");
currentPhase = "load_spam_circle_cache";
var spamStopwatch = Stopwatch.StartNew();
string[] excludedMakerIds = await spamCircleCache.GetAsync(cancellationToken);
spamStopwatch.Stop();
logger.LogInformation(
"Loaded spam circle cache in {ElapsedMs} ms. ExcludedMakerCount={ExcludedMakerCount}",
spamStopwatch.ElapsedMilliseconds,
excludedMakerIds.Length);
VoiceWorkScanOptions options = new( VoiceWorkScanOptions options = new(
PageNumber: request.PageNumber, PageNumber: request.PageNumber,
PageSize: request.PageSize, PageSize: request.PageSize,
ExcludedMakerIds: await spamCircleCache.GetAsync(cancellationToken), ExcludedMakerIds: excludedMakerIds,
ExcludePartiallyAIGeneratedWorks: true, ExcludePartiallyAIGeneratedWorks: true,
ExcludeAIGeneratedWorks: true ExcludeAIGeneratedWorks: true
); );
IReadOnlyList<DLSiteWork> works = await scanner.ScanPageAsync(options, cancellationToken); currentPhase = "scan_page";
var scanStopwatch = Stopwatch.StartNew();
if (works.Count == 0) VoiceWorkScanResult scanResult = await scanner.ScanPageAsync(options, cancellationToken);
return new();
string[] productIds = [.. works.Where(x => !string.IsNullOrWhiteSpace(x.ProductId)).Select(x => x.ProductId!)]; scanStopwatch.Stop();
VoiceWorkDetailCollection voiceWorkDetails = await dlsiteClient.GetVoiceWorkDetailsAsync(productIds, cancellationToken);
VoiceWorkIngest[] ingests = [.. works.Select(work => logger.LogInformation(
"Scanned source page in {ElapsedMs} ms. EndOfResults={EndOfResults}, ResultCount={ResultCount}",
scanStopwatch.ElapsedMilliseconds,
scanResult.EndOfResults,
scanResult.Works.Length);
if (scanResult.EndOfResults)
{ {
voiceWorkDetails.TryGetValue(work.ProductId!, out VoiceWorkDetails? value); stopwatch.Stop();
return VoiceWorkIngest.From(work, value);
})]; logger.LogInformation(
"End of results reached for page {PageNumber}. TotalElapsedMs={ElapsedMs}",
request.PageNumber,
stopwatch.ElapsedMilliseconds);
return new ScanVoiceWorksResponse(
Results: [],
EndOfResults: true
);
}
currentPhase = "build_ingests";
var ingestSw = Stopwatch.StartNew();
VoiceWorkIngest[] ingests = await ingestBuilder.BuildAsync(scanResult, cancellationToken);
ingestSw.Stop();
logger.LogInformation(
"Built ingests in {ElapsedMs} ms. IngestCount={IngestCount}",
ingestSw.ElapsedMilliseconds,
ingests.Length);
currentPhase = "upsert_voiceworks";
var upsertSw = Stopwatch.StartNew();
VoiceWorkUpsertResult[] upsertResults = await updater.UpsertAsync(ingests, cancellationToken); VoiceWorkUpsertResult[] upsertResults = await updater.UpsertAsync(ingests, cancellationToken);
upsertSw.Stop();
int[] voiceWorkIds = [.. upsertResults.Where(x => x.VoiceWorkId.HasValue).Select(x => x.VoiceWorkId!.Value)]; int[] voiceWorkIds = [.. upsertResults.Where(x => x.VoiceWorkId.HasValue).Select(x => x.VoiceWorkId!.Value)];
int issueCount = upsertResults.Sum(x => x.Issues.Count);
logger.LogInformation(
"Upserted voice works in {ElapsedMs} ms. UpsertResultCount={UpsertResultCount}, VoiceWorkIdCount={VoiceWorkIdCount}, IssueCount={IssueCount}",
upsertSw.ElapsedMilliseconds,
upsertResults.Length,
voiceWorkIds.Length,
issueCount);
currentPhase = "update_search";
var searchSw = Stopwatch.StartNew();
await searchUpdater.UpdateAsync(voiceWorkIds, cancellationToken); await searchUpdater.UpdateAsync(voiceWorkIds, cancellationToken);
return new(); searchSw.Stop();
logger.LogInformation(
"Updated search index in {ElapsedMs} ms. VoiceWorkIdCount={VoiceWorkIdCount}",
searchSw.ElapsedMilliseconds,
voiceWorkIds.Length);
stopwatch.Stop();
logger.LogInformation(
"Completed scan handler for page {PageNumber} in {ElapsedMs} ms",
request.PageNumber,
stopwatch.ElapsedMilliseconds);
return new ScanVoiceWorksResponse(
Results: upsertResults,
EndOfResults: false
);
}
catch (Exception ex)
{
stopwatch.Stop();
logger.LogError(
ex,
"Scan handler failed during phase {Phase} for page {PageNumber} after {ElapsedMs} ms",
currentPhase,
request.PageNumber,
stopwatch.ElapsedMilliseconds);
throw;
}
} }
} }

View File

@@ -2,4 +2,4 @@
namespace JSMR.Application.Scanning; namespace JSMR.Application.Scanning;
public sealed record ScanVoiceWorksRequest(int PageNumber, int PageSize, Locale Locale, string[] ExcludedMakerIds); public sealed record ScanVoiceWorksRequest(int PageNumber, int PageSize, Locale Locale);

View File

@@ -1,7 +1,8 @@
namespace JSMR.Application.Scanning; using JSMR.Application.Scanning.Ports;
public sealed class ScanVoiceWorksResponse namespace JSMR.Application.Scanning;
{
public int Inserted { get; init; } public sealed record ScanVoiceWorksResponse(
public int Updated { get; init; } VoiceWorkUpsertResult[] Results,
} bool EndOfResults
);

View File

@@ -2,10 +2,10 @@
public record TagSearchItem public record TagSearchItem
{ {
public int TagId { get; init; } public int TagId { get; set; }
public required string Name { get; init; } public required string Name { get; set; }
public bool Favorite { get; init; } public bool Favorite { get; set; }
public bool Blacklisted { get; init; } public bool Blacklisted { get; set; }
public string? EnglishName { get; init; } public string? EnglishName { get; set; }
public int VoiceWorkCount { get; init; } public int VoiceWorkCount { get; set; }
} }

View File

@@ -0,0 +1,9 @@
using JSMR.Domain.Entities;
namespace JSMR.Application.Users;
public interface IUserRepository
{
Task<User?> FindByUsernameAsync(string username);
bool VerifyPassword(User user, string password);
}

View File

@@ -0,0 +1,11 @@
using JSMR.Application.VoiceWorks.Ports;
namespace JSMR.Application.VoiceWorks.Commands.Delete;
public sealed class DeleteVoiceWorkFavoriteHandler(IVoiceWorkWriter writer)
{
public async Task<DeleteVoiceWorkResponse> HandleAsync(DeleteVoiceWorkRequest request, CancellationToken cancellationToken = default)
{
return await writer.DeleteAsync(request, cancellationToken);
}
}

View File

@@ -0,0 +1,3 @@
namespace JSMR.Application.VoiceWorks.Commands.Delete;
public sealed record DeleteVoiceWorkRequest(int[] VoiceWorkIds);

View File

@@ -0,0 +1,3 @@
namespace JSMR.Application.VoiceWorks.Commands.Delete;
public sealed record DeleteVoiceWorkResponse(Dictionary<int, bool> IsSuccess);

View File

@@ -1,8 +1,10 @@
using JSMR.Application.VoiceWorks.Commands.SetFavorite; using JSMR.Application.VoiceWorks.Commands.Delete;
using JSMR.Application.VoiceWorks.Commands.SetFavorite;
namespace JSMR.Application.VoiceWorks.Ports; namespace JSMR.Application.VoiceWorks.Ports;
public interface IVoiceWorkWriter public interface IVoiceWorkWriter
{ {
Task<SetVoiceWorkFavoriteResponse> SetFavoriteAsync(SetVoiceWorkFavoriteRequest request, CancellationToken cancellationToken); Task<SetVoiceWorkFavoriteResponse> SetFavoriteAsync(SetVoiceWorkFavoriteRequest request, CancellationToken cancellationToken);
Task<DeleteVoiceWorkResponse> DeleteAsync(DeleteVoiceWorkRequest request, CancellationToken cancellationToken);
} }

View File

@@ -27,18 +27,33 @@ public record VoiceWorkSearchResult
public byte Status { get; init; } public byte Status { get; init; }
public byte SubtitleLanguage { get; init; } public byte SubtitleLanguage { get; init; }
public bool? IsValid { get; init; } public bool? IsValid { get; init; }
public required VoiceWorkCircleItem Circle { get; set; }
public VoiceWorkCircleItem? OriginalCircle { get; set; }
public VoiceWorkTagItem[] Tags { get; set; } = []; public VoiceWorkTagItem[] Tags { get; set; } = [];
public VoiceWorkCreatorItem[] Creators { get; set; } = []; public VoiceWorkCreatorItem[] Creators { get; set; } = [];
public string[] SupportedLanguages { get; set; } = [];
}
public class VoiceWorkCircleItem
{
public required string Name { get; init; }
public required string MakerId { get; init; }
public bool IsFavorite { get; init; }
public bool IsBlacklisted { get; init; }
} }
public class VoiceWorkTagItem public class VoiceWorkTagItem
{ {
public int TagId { get; set; } public int TagId { get; set; }
public required string Name { get; set; } public required string Name { get; set; }
public bool IsFavorite { get; set; }
public bool IsBlacklisted { get; set; }
} }
public class VoiceWorkCreatorItem public class VoiceWorkCreatorItem
{ {
public int CreatorId { get; set; } public int CreatorId { get; set; }
public required string Name { get; set; } public required string Name { get; set; }
public bool IsFavorite { get; set; }
public bool IsBlacklisted { get; set; }
} }

View File

@@ -0,0 +1,33 @@
using JSMR.Domain.Enums;
namespace JSMR.Domain.Entities;
public sealed class Job
{
public int Id { get; set; }
public string Code { get; set; } = null!;
public JobStatus Status { get; set; }
public string? RequestedByUserId { get; set; }
public string RequestedSource { get; set; } = "Manual";
public string? ParametersJson { get; set; }
public DateTime CreatedUtc { get; set; }
public DateTime? StartedUtc { get; set; }
public DateTime? CompletedUtc { get; set; }
public DateTime? HeartbeatUtc { get; set; }
public string? WorkerName { get; set; }
public string? CurrentStep { get; set; }
public int? ProgressCurrent { get; set; }
public int? ProgressTotal { get; set; }
public string? ResultSummary { get; set; }
public string? Error { get; set; }
public bool CancellationRequested { get; set; }
public int AttemptCount { get; set; }
}

View File

@@ -0,0 +1,10 @@
namespace JSMR.Domain.Entities;
public class User
{
public int Id { get; set; }
public string Username { get; set; } = string.Empty;
public string PasswordHash { get; set; } = string.Empty;
public string? Role { get; set; }
public bool IsActive { get; set; } = true;
}

View File

@@ -0,0 +1,10 @@
namespace JSMR.Domain.Enums;
public enum JobStatus
{
Queued = 0,
Running = 1,
Succeeded = 2,
Failed = 3,
Cancelled = 4
}

View File

@@ -1,7 +1,7 @@
<Project Sdk="Microsoft.NET.Sdk"> <Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup> <PropertyGroup>
<TargetFramework>net9.0</TargetFramework> <TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings> <ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable> <Nullable>enable</Nullable>
</PropertyGroup> </PropertyGroup>

View File

@@ -5,9 +5,13 @@ using JSMR.Application.Common.Caching;
using JSMR.Application.Creators.Ports; using JSMR.Application.Creators.Ports;
using JSMR.Application.Creators.Queries.Search.Ports; using JSMR.Application.Creators.Queries.Search.Ports;
using JSMR.Application.Enums; using JSMR.Application.Enums;
using JSMR.Application.Integrations.Chobit.Ports;
using JSMR.Application.Integrations.DLSite.Ports;
using JSMR.Application.Jobs;
using JSMR.Application.Scanning.Ports; using JSMR.Application.Scanning.Ports;
using JSMR.Application.Tags.Ports; using JSMR.Application.Tags.Ports;
using JSMR.Application.Tags.Queries.Search.Ports; using JSMR.Application.Tags.Queries.Search.Ports;
using JSMR.Application.Users;
using JSMR.Application.VoiceWorks.Ports; using JSMR.Application.VoiceWorks.Ports;
using JSMR.Application.VoiceWorks.Queries.Search; using JSMR.Application.VoiceWorks.Queries.Search;
using JSMR.Infrastructure.Caching; using JSMR.Infrastructure.Caching;
@@ -17,12 +21,19 @@ using JSMR.Infrastructure.Common.SupportedLanguages;
using JSMR.Infrastructure.Common.Time; using JSMR.Infrastructure.Common.Time;
using JSMR.Infrastructure.Data.Repositories.Circles; using JSMR.Infrastructure.Data.Repositories.Circles;
using JSMR.Infrastructure.Data.Repositories.Creators; using JSMR.Infrastructure.Data.Repositories.Creators;
using JSMR.Infrastructure.Data.Repositories.Jobs;
using JSMR.Infrastructure.Data.Repositories.Tags; using JSMR.Infrastructure.Data.Repositories.Tags;
using JSMR.Infrastructure.Data.Repositories.Users;
using JSMR.Infrastructure.Data.Repositories.VoiceWorks; using JSMR.Infrastructure.Data.Repositories.VoiceWorks;
using JSMR.Infrastructure.Http; using JSMR.Infrastructure.Http;
using JSMR.Infrastructure.Ingestion; using JSMR.Infrastructure.Ingestion;
using JSMR.Infrastructure.Integrations.Chobit;
using JSMR.Infrastructure.Integrations.DLSite;
using JSMR.Infrastructure.Scanning; using JSMR.Infrastructure.Scanning;
using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http.Resilience;
using Polly;
using System.Net;
namespace JSMR.Infrastructure.DI; namespace JSMR.Infrastructure.DI;
@@ -40,9 +51,16 @@ public static class InfrastructureServiceCollectionExtensions
services.AddKeyedScoped<IVoiceWorksScanner, JapaneseVoiceWorksScanner>(Locale.Japanese); services.AddKeyedScoped<IVoiceWorksScanner, JapaneseVoiceWorksScanner>(Locale.Japanese);
services.AddKeyedScoped<IVoiceWorksScanner, EnglishVoiceWorksScanner>(Locale.English); services.AddKeyedScoped<IVoiceWorksScanner, EnglishVoiceWorksScanner>(Locale.English);
services.AddScoped<IVoiceWorkScannerRepository, VoiceWorkScannerRepository>();
services.AddScoped<IReleasedWorksProvider, ReleasedWorksProvider>();
services.AddScoped<IVoiceWorkIngestBuilder, VoiceWorkIngestBuilder>();
services.AddKeyedScoped<IVoiceWorkUpdater, VoiceWorkUpdater>(Locale.Japanese); services.AddKeyedScoped<IVoiceWorkUpdater, VoiceWorkUpdater>(Locale.Japanese);
services.AddKeyedScoped<IVoiceWorkUpdater, EnglishVoiceWorkUpdater>(Locale.English); services.AddKeyedScoped<IVoiceWorkUpdater, EnglishVoiceWorkUpdater>(Locale.English);
services.AddScoped<IVoiceWorkUpdaterRepository, VoiceWorkUpdaterRepository>();
services.AddScoped<IVoiceWorkSearchUpdater, VoiceWorkSearchUpdater>();
//services.AddKeyedScoped<ISupportedLanguage, JapaneseLanguage>(Locale.Japanese); //services.AddKeyedScoped<ISupportedLanguage, JapaneseLanguage>(Locale.Japanese);
//services.AddKeyedScoped<ISupportedLanguage, EnglishLanguage>(Locale.English); //services.AddKeyedScoped<ISupportedLanguage, EnglishLanguage>(Locale.English);
@@ -56,45 +74,103 @@ public static class InfrastructureServiceCollectionExtensions
services.AddScoped<ICreatorSearchProvider, CreatorSearchProvider>(); services.AddScoped<ICreatorSearchProvider, CreatorSearchProvider>();
services.AddScoped<ICreatorWriter, CreatorWriter>(); services.AddScoped<ICreatorWriter, CreatorWriter>();
services.AddScoped<IJobRepository, JobRepository>();
services.AddScoped<IJobProgressWriter, JobProgressWriter>();
services.AddSingleton<ICache, MemoryCacheAdapter>(); services.AddSingleton<ICache, MemoryCacheAdapter>();
services.AddSingleton<ISpamCircleCache, SpamCircleCache>(); services.AddSingleton<ISpamCircleCache, SpamCircleCache>();
services.AddHttpClient<IHttpService, HttpService>(client =>
{
client.DefaultRequestHeaders.UserAgent.ParseAdd("JSMR/1.0");
});
services.AddScoped<IHttpService, HttpService>();
services.AddScoped<IHtmlLoader, HtmlLoader>();
services.AddSingleton<ILanguageIdentifier, LanguageIdentifier>(); services.AddSingleton<ILanguageIdentifier, LanguageIdentifier>();
services.AddSingleton<IClock, Clock>(); services.AddSingleton<IClock, Clock>();
services.AddSingleton<ITimeProvider, TokyoTimeProvider>(); services.AddSingleton<ITimeProvider, TokyoTimeProvider>();
services.AddHttpServices();
services.AddNewHttpServices();
services.AddScoped<IUserRepository, UserRepository>();
return services; return services;
} }
//public static IServiceCollection AddDLSiteClient(this IServiceCollection services) private static IServiceCollection AddHttpServices(this IServiceCollection services)
{
//services.AddHttpClient<IHttpService, HttpService>(client =>
//{ //{
// var retryPolicy = HttpPolicyExtensions // client.DefaultRequestHeaders.UserAgent.ParseAdd("JSMR/1.0");
// .HandleTransientHttpError() //});
// .OrResult(msg => (int)msg.StatusCode == 429) // Too Many Requests
// .WaitAndRetryAsync(new[]
// {
// TimeSpan.FromMilliseconds(200),
// TimeSpan.FromMilliseconds(500),
// TimeSpan.FromSeconds(1.5)
// });
// services.AddHttpClient<IDLSiteClient, DLSiteClient>(c => //services.AddScoped<IHttpService, HttpService>();
// { services.AddScoped<IHtmlLoader, HtmlLoader>();
// c.BaseAddress = new Uri("https://www.dlsite.com/");
// c.DefaultRequestHeaders.UserAgent.ParseAdd("JSMR/1.0 (+contact@example.com)");
// c.Timeout = TimeSpan.FromSeconds(15);
// })
// .AddPolicyHandler(retryPolicy);
// return services; // ONE registration for IHttpService as a typed client:
//} services.AddHttpClient<IHttpService, HttpService>((sp, http) =>
{
http.BaseAddress = new Uri("https://www.dlsite.com/");
http.DefaultRequestHeaders.UserAgent.ParseAdd("JSMR/1.0");
http.Timeout = TimeSpan.FromSeconds(15);
})
.AddResilienceHandler("dlsite", builder => {
builder.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 3,
UseJitter = true,
Delay = TimeSpan.FromMilliseconds(200),
BackoffType = DelayBackoffType.Exponential,
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.Handle<HttpRequestException>()
.HandleResult(r => (int)r.StatusCode >= 500 || (int)r.StatusCode == 429)
});
});
// Register DLSiteClient as a normal scoped service
services.AddScoped<IDLSiteClient, DLSiteClient>();
return services;
}
private static IServiceCollection AddNewHttpServices(this IServiceCollection services)
{
services.AddHttpClient<IDLSiteClient, DLSiteClient>((sp, http) =>
{
http.BaseAddress = new Uri("https://www.dlsite.com/");
http.DefaultRequestHeaders.UserAgent.ParseAdd("JSMR/1.0");
http.Timeout = TimeSpan.FromSeconds(15);
})
.AddResilienceHandler("dlsite", builder =>
{
builder.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 3,
UseJitter = true,
Delay = TimeSpan.FromMilliseconds(200),
BackoffType = DelayBackoffType.Exponential,
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.Handle<HttpRequestException>()
.HandleResult(r => (int)r.StatusCode >= 500 || (int)r.StatusCode == 429)
});
});
services.AddHttpClient<IChobitClient, ChobitClient>((sp, http) =>
{
http.BaseAddress = new Uri("https://chobit.cc/");
http.DefaultRequestHeaders.UserAgent.ParseAdd("JSMR/1.0");
http.Timeout = TimeSpan.FromSeconds(15);
})
.AddResilienceHandler("chobit", builder =>
{
builder.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 3,
UseJitter = true,
Delay = TimeSpan.FromMilliseconds(200),
BackoffType = DelayBackoffType.Exponential,
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.Handle<HttpRequestException>()
.HandleResult(r => (int)r.StatusCode >= 500 || (int)r.StatusCode == 429)
});
});
return services;
}
} }

View File

@@ -17,6 +17,8 @@ public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(op
public DbSet<VoiceWorkCreator> VoiceWorkCreators { get; set; } public DbSet<VoiceWorkCreator> VoiceWorkCreators { get; set; }
public DbSet<Series> Series { get; set; } public DbSet<Series> Series { get; set; }
public DbSet<VoiceWorkSearch> VoiceWorkSearches { get; set; } public DbSet<VoiceWorkSearch> VoiceWorkSearches { get; set; }
public DbSet<User> Users { get; set; }
public DbSet<Job> Jobs { get; set; }
protected override void OnModelCreating(ModelBuilder modelBuilder) protected override void OnModelCreating(ModelBuilder modelBuilder)
{ {

View File

@@ -0,0 +1,30 @@
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Design;
using Microsoft.Extensions.Configuration;
namespace JSMR.Infrastructure.Data;
public sealed class AppDbContextFactory : IDesignTimeDbContextFactory<AppDbContext>
{
public AppDbContext CreateDbContext(string[] args)
{
// adjust base path if needed (points to the worker for secrets/env)
var config = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: true)
.AddJsonFile("appsettings.Development.json", optional: true)
.AddUserSecrets(typeof(AppDbContextFactory).Assembly, optional: true)
.AddEnvironmentVariables()
.Build();
var conn = config.GetConnectionString("AppDb")
?? throw new InvalidOperationException("Missing ConnectionStrings:AppDb");
var options = new DbContextOptionsBuilder<AppDbContext>()
.UseMySql(conn, ServerVersion.AutoDetect(conn))
.EnableSensitiveDataLogging(false)
.Options;
return new AppDbContext(options);
}
}

View File

@@ -0,0 +1,44 @@
using JSMR.Domain.Entities;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Metadata.Builders;
namespace JSMR.Infrastructure.Data.Configuration;
public sealed class JobConfiguration : IEntityTypeConfiguration<Job>
{
public void Configure(EntityTypeBuilder<Job> builder)
{
builder.ToTable("Jobs");
builder.HasKey(x => x.Id);
builder.Property(x => x.Code)
.HasMaxLength(100)
.IsRequired();
builder.Property(x => x.RequestedByUserId)
.HasMaxLength(100);
builder.Property(x => x.RequestedSource)
.HasMaxLength(50)
.IsRequired();
builder.Property(x => x.WorkerName)
.HasMaxLength(200);
builder.Property(x => x.CurrentStep)
.HasMaxLength(500);
builder.Property(x => x.ResultSummary)
.HasMaxLength(2000);
builder.Property(x => x.Error)
.HasColumnType("TEXT");
builder.Property(x => x.ParametersJson)
.HasColumnType("LONGTEXT");
builder.HasIndex(x => new { x.Status, x.CreatedUtc });
builder.HasIndex(x => x.Code);
}
}

View File

@@ -8,7 +8,7 @@ public sealed class VoiceWorkLocalizationConfiguration : IEntityTypeConfiguratio
{ {
public void Configure(EntityTypeBuilder<VoiceWorkLocalization> builder) public void Configure(EntityTypeBuilder<VoiceWorkLocalization> builder)
{ {
builder.ToTable("voice_work_localizations"); builder.ToTable("VoiceWorkLocalizations");
builder.HasKey(x => x.VoiceWorkLocalizationId); builder.HasKey(x => x.VoiceWorkLocalizationId);
builder.Property(x => x.Language).IsRequired().HasMaxLength(10); builder.Property(x => x.Language).IsRequired().HasMaxLength(10);

View File

@@ -8,7 +8,7 @@ public sealed class VoiceWorkSearchConfiguration : IEntityTypeConfiguration<Voic
{ {
public void Configure(EntityTypeBuilder<VoiceWorkSearch> builder) public void Configure(EntityTypeBuilder<VoiceWorkSearch> builder)
{ {
builder.ToTable("VoiceWorkSearches2"); builder.ToTable("VoiceWorkSearches");
builder.HasKey(x => x.VoiceWorkId); // also the FK builder.HasKey(x => x.VoiceWorkId); // also the FK
builder.Property(x => x.SearchText).IsRequired(); // TEXT/LONGTEXT builder.Property(x => x.SearchText).IsRequired(); // TEXT/LONGTEXT

View File

@@ -8,7 +8,7 @@ public sealed class VoiceWorkSupportedLanguageConfiguration : IEntityTypeConfigu
{ {
public void Configure(EntityTypeBuilder<VoiceWorkSupportedLanguage> builder) public void Configure(EntityTypeBuilder<VoiceWorkSupportedLanguage> builder)
{ {
builder.ToTable("voice_work_supported_languages"); builder.ToTable("VoiceWorkSupportedLanguages");
builder.HasKey(x => x.VoiceWorkSupportedLanguageId); builder.HasKey(x => x.VoiceWorkSupportedLanguageId);
builder.Property(x => x.Language).IsRequired().HasMaxLength(10); builder.Property(x => x.Language).IsRequired().HasMaxLength(10);

View File

@@ -29,69 +29,6 @@ public class CircleSearchProvider(AppDbContext context) : SearchProvider<CircleS
return q; return q;
} }
//protected override IQueryable<CircleSearchItem> GetBaseQuery()
//{
// // Project from Circles so we can use correlated subqueries per CircleId.
// var q =
// from c in context.Circles.AsNoTracking()
// select new CircleSearchItem
// {
// CircleId = c.CircleId,
// Name = c.Name,
// MakerId = c.MakerId,
// Favorite = c.Favorite,
// Blacklisted = c.Blacklisted,
// Spam = c.Spam,
// // Aggregates
// Downloads = context.VoiceWorks
// .Where(v => v.CircleId == c.CircleId)
// .Select(v => (int?)v.Downloads) // make nullable for Sum over empty set
// .Sum() ?? 0,
// Releases = context.VoiceWorks
// .Count(v => v.CircleId == c.CircleId && v.SalesDate != null),
// Pending = context.VoiceWorks
// .Count(v => v.CircleId == c.CircleId && v.ExpectedDate != null),
// FirstReleaseDate = context.VoiceWorks
// .Where(v => v.CircleId == c.CircleId)
// .Select(v => v.SalesDate)
// .Min(),
// LatestReleaseDate = context.VoiceWorks
// .Where(v => v.CircleId == c.CircleId)
// .Select(v => v.SalesDate)
// .Max(),
// // "Latest" by ProductId length, then value
// LatestProductId = context.VoiceWorks
// .Where(v => v.CircleId == c.CircleId)
// .OrderByDescending(v => v.ProductId.Length)
// .ThenByDescending(v => v.ProductId)
// .Select(v => v.ProductId)
// .FirstOrDefault(),
// // If you want these two in base query too:
// LatestVoiceWorkHasImage = context.VoiceWorks
// .Where(v => v.CircleId == c.CircleId)
// .OrderByDescending(v => v.ProductId.Length)
// .ThenByDescending(v => v.ProductId)
// .Select(v => (bool?)v.HasImage)
// .FirstOrDefault(),
// LatestVoiceWorkSalesDate = context.VoiceWorks
// .Where(v => v.CircleId == c.CircleId)
// .OrderByDescending(v => v.ProductId.Length)
// .ThenByDescending(v => v.ProductId)
// .Select(v => v.SalesDate)
// .FirstOrDefault()
// };
// return q;
//}
protected override IQueryable<CircleQuery> ApplyFilters(IQueryable<CircleQuery> query, CircleSearchCriteria criteria) protected override IQueryable<CircleQuery> ApplyFilters(IQueryable<CircleQuery> query, CircleSearchCriteria criteria)
{ {
if (!string.IsNullOrWhiteSpace(criteria.Name)) if (!string.IsNullOrWhiteSpace(criteria.Name))

View File

@@ -44,8 +44,8 @@ public class CreatorSearchProvider(AppDbContext context) : SearchProvider<Creato
Expression<Func<CreatorSearchItem, object?>> selector = field switch Expression<Func<CreatorSearchItem, object?>> selector = field switch
{ {
CreatorSortField.VoiceWorkCount => x => x.VoiceWorkCount, CreatorSortField.VoiceWorkCount => x => x.VoiceWorkCount,
CreatorSortField.Favorite => x => !x.Favorite, CreatorSortField.Favorite => x => x.Favorite,
CreatorSortField.Blacklisted => x => !x.Blacklisted, CreatorSortField.Blacklisted => x => x.Blacklisted,
_ => x => x.Name _ => x => x.Name
}; };

View File

@@ -0,0 +1,80 @@
using JSMR.Application.Jobs;
using JSMR.Domain.Entities;
using JSMR.Domain.Enums;
using Microsoft.EntityFrameworkCore;
namespace JSMR.Infrastructure.Data.Repositories.Jobs;
public sealed class JobProgressWriter(AppDbContext dbContext) : IJobProgressWriter
{
public async Task SetStepAsync(int jobId, string step, CancellationToken canellationToken)
{
Job? job = await dbContext.Jobs.FirstOrDefaultAsync(x => x.Id == jobId, canellationToken);
if (job is null)
return;
job.CurrentStep = step;
job.HeartbeatUtc = DateTime.UtcNow;
await dbContext.SaveChangesAsync(canellationToken);
}
public async Task SetProgressAsync(int jobId, int? current, int? total, CancellationToken canellationToken)
{
Job? job = await dbContext.Jobs.FirstOrDefaultAsync(x => x.Id == jobId, canellationToken);
if (job is null)
return;
job.ProgressCurrent = current;
job.ProgressTotal = total;
job.HeartbeatUtc = DateTime.UtcNow;
await dbContext.SaveChangesAsync(canellationToken);
}
public async Task SetHeartbeatAsync(int jobId, CancellationToken canellationToken)
{
Job? job = await dbContext.Jobs.FirstOrDefaultAsync(x => x.Id == jobId, canellationToken);
if (job is null)
return;
job.HeartbeatUtc = DateTime.UtcNow;
await dbContext.SaveChangesAsync(canellationToken);
}
public async Task CompleteAsync(int jobId, string? summary, CancellationToken canellationToken)
{
Job? job = await dbContext.Jobs.FirstOrDefaultAsync(x => x.Id == jobId, canellationToken);
if (job is null)
return;
job.Status = JobStatus.Succeeded;
job.CompletedUtc = DateTime.UtcNow;
job.HeartbeatUtc = DateTime.UtcNow;
job.ResultSummary = summary;
job.CurrentStep = "Completed";
await dbContext.SaveChangesAsync(canellationToken);
}
public async Task FailAsync(int jobId, string error, CancellationToken canellationToken)
{
Job? job = await dbContext.Jobs.FirstOrDefaultAsync(x => x.Id == jobId, canellationToken);
if (job is null)
return;
job.Status = JobStatus.Failed;
job.CompletedUtc = DateTime.UtcNow;
job.HeartbeatUtc = DateTime.UtcNow;
job.Error = error;
job.CurrentStep = "Failed";
await dbContext.SaveChangesAsync(canellationToken);
}
}

View File

@@ -0,0 +1,52 @@
using JSMR.Application.Jobs;
using JSMR.Domain.Entities;
using JSMR.Domain.Enums;
using Microsoft.EntityFrameworkCore;
namespace JSMR.Infrastructure.Data.Repositories.Jobs;
public sealed class JobRepository(AppDbContext dbContext) : IJobRepository
{
public async Task<Job> AddAsync(Job job, CancellationToken cancellationToken)
{
await dbContext.Jobs.AddAsync(job, cancellationToken);
return job;
}
public Task<Job?> GetByIdAsync(int id, CancellationToken cancellationToken)
=> dbContext.Jobs.FirstOrDefaultAsync(x => x.Id == id, cancellationToken);
public async Task<IReadOnlyList<Job>> GetRecentAsync(int take, CancellationToken cancellationToken)
=> await dbContext.Jobs
.OrderByDescending(x => x.CreatedUtc)
.Take(take)
.ToListAsync(cancellationToken);
public Task<bool> AnyRunningAsync(CancellationToken cancellationToken)
=> dbContext.Jobs.AnyAsync(x => x.Status == JobStatus.Running, cancellationToken);
public async Task<Job?> TryClaimNextQueuedAsync(string workerName, CancellationToken cancellationToken)
{
Job? next = await dbContext.Jobs
.Where(x => x.Status == JobStatus.Queued)
.OrderBy(x => x.CreatedUtc)
.FirstOrDefaultAsync(cancellationToken);
if (next is null)
return null;
next.Status = JobStatus.Running;
next.StartedUtc = DateTime.UtcNow;
next.HeartbeatUtc = DateTime.UtcNow;
next.WorkerName = workerName;
next.AttemptCount += 1;
await dbContext.SaveChangesAsync(cancellationToken);
return next;
}
public Task SaveChangesAsync(CancellationToken ct)
=> dbContext.SaveChangesAsync(ct);
}

View File

@@ -0,0 +1,20 @@
using JSMR.Application.Users;
using JSMR.Domain.Entities;
using Microsoft.EntityFrameworkCore;
namespace JSMR.Infrastructure.Data.Repositories.Users;
public class UserRepository(AppDbContext db) : IUserRepository
{
public async Task<User?> FindByUsernameAsync(string username)
{
return await db.Users
.FirstOrDefaultAsync(u => u.Username == username);
}
public bool VerifyPassword(User user, string password)
{
// Using BCrypt (recommended)
return BCrypt.Net.BCrypt.Verify(password, user.PasswordHash);
}
}

View File

@@ -3,6 +3,7 @@ using JSMR.Application.Enums;
using JSMR.Application.VoiceWorks.Queries.Search; using JSMR.Application.VoiceWorks.Queries.Search;
using JSMR.Domain.Entities; using JSMR.Domain.Entities;
using JSMR.Domain.Enums; using JSMR.Domain.Enums;
using JSMR.Domain.ValueObjects;
using JSMR.Infrastructure.Common.Queries; using JSMR.Infrastructure.Common.Queries;
using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore;
using System.Linq.Expressions; using System.Linq.Expressions;
@@ -60,9 +61,10 @@ public class VoiceWorkSearchProvider(AppDbContext context, IVoiceWorkFullTextSea
break; break;
} }
if (criteria.SupportedLanguages.Length > 0) //if (criteria.SupportedLanguages.Length > 0)
filteredQuery = filteredQuery.Where(x => criteria.SupportedLanguages.Contains((Language)x.VoiceWork.SubtitleLanguage)); // filteredQuery = filteredQuery.Where(x => criteria.SupportedLanguages.AsEnumerable().Contains((Language)x.VoiceWork.SubtitleLanguage));
filteredQuery = ApplySupportedLanguageFilter(filteredQuery, criteria);
filteredQuery = ApplyCircleStatusFilter(filteredQuery, criteria); filteredQuery = ApplyCircleStatusFilter(filteredQuery, criteria);
filteredQuery = ApplyTagStatusFilter(filteredQuery, criteria); filteredQuery = ApplyTagStatusFilter(filteredQuery, criteria);
filteredQuery = ApplyCreatorStatusFilter(filteredQuery, criteria); filteredQuery = ApplyCreatorStatusFilter(filteredQuery, criteria);
@@ -85,13 +87,13 @@ public class VoiceWorkSearchProvider(AppDbContext context, IVoiceWorkFullTextSea
filteredQuery = filteredQuery.Where(x => x.VoiceWork.SalesDate <= criteria.ReleaseDateEnd.Value.ToDateTime(TimeOnly.MinValue)); filteredQuery = filteredQuery.Where(x => x.VoiceWork.SalesDate <= criteria.ReleaseDateEnd.Value.ToDateTime(TimeOnly.MinValue));
if (criteria.AgeRatings.Length > 0) if (criteria.AgeRatings.Length > 0)
filteredQuery = filteredQuery.Where(x => criteria.AgeRatings.Contains((AgeRating)x.VoiceWork.Rating)); filteredQuery = filteredQuery.Where(x => criteria.AgeRatings.AsEnumerable().Contains((AgeRating)x.VoiceWork.Rating));
//if (criteria.SupportedLanguages.Length > 0) //if (criteria.SupportedLanguages.Length > 0)
// filteredQuery = filteredQuery.Where(x => criteria.SupportedLanguages.Contains((Language)x.VoiceWork.SubtitleLanguage)); // filteredQuery = filteredQuery.Where(x => criteria.SupportedLanguages.Contains((Language)x.VoiceWork.SubtitleLanguage));
if (criteria.AIGenerationOptions.Length > 0) if (criteria.AIGenerationOptions.Length > 0)
filteredQuery = filteredQuery.Where(x => criteria.AIGenerationOptions.Contains((AIGeneration)x.VoiceWork.AIGeneration)); filteredQuery = filteredQuery.Where(x => criteria.AIGenerationOptions.AsEnumerable().Contains((AIGeneration)x.VoiceWork.AIGeneration));
if (criteria.ShowFavoriteVoiceWorks) if (criteria.ShowFavoriteVoiceWorks)
filteredQuery = filteredQuery.Where(x => x.VoiceWork.Favorite); filteredQuery = filteredQuery.Where(x => x.VoiceWork.Favorite);
@@ -118,6 +120,26 @@ public class VoiceWorkSearchProvider(AppDbContext context, IVoiceWorkFullTextSea
return query.Where(x => voiceWorkIds.Contains(x.VoiceWork.VoiceWorkId)); return query.Where(x => voiceWorkIds.Contains(x.VoiceWork.VoiceWorkId));
} }
private IQueryable<VoiceWorkQuery> ApplySupportedLanguageFilter(IQueryable<VoiceWorkQuery> query, VoiceWorkSearchCriteria criteria)
{
if (criteria.SupportedLanguages.Length == 0)
return query;
List<string> languageCodes = [];
foreach (Language language in criteria.SupportedLanguages)
{
if (SupportedLanguage.TryFromLanguage(language, out SupportedLanguage? supportedLanguage))
languageCodes.Add(supportedLanguage.Code);
}
if (languageCodes.Count == 0)
return query;
return query.Where(q => context.VoiceWorkSupportedLanguages.Any(sl => sl.VoiceWorkId == q.VoiceWork.VoiceWorkId && languageCodes.Contains(sl.Language)));
}
private IQueryable<VoiceWorkQuery> ApplyCircleStatusFilter(IQueryable<VoiceWorkQuery> query, VoiceWorkSearchCriteria criteria) private IQueryable<VoiceWorkQuery> ApplyCircleStatusFilter(IQueryable<VoiceWorkQuery> query, VoiceWorkSearchCriteria criteria)
{ {
if (criteria.CircleStatus is null) if (criteria.CircleStatus is null)
@@ -376,6 +398,13 @@ public class VoiceWorkSearchProvider(AppDbContext context, IVoiceWorkFullTextSea
HasImage = voiceWork.HasImage, HasImage = voiceWork.HasImage,
Maker = circle.Name, Maker = circle.Name,
MakerId = circle.MakerId, MakerId = circle.MakerId,
Circle = new()
{
Name = circle.Name,
MakerId = circle.MakerId,
IsFavorite = circle.Favorite,
IsBlacklisted = circle.Blacklisted
},
ExpectedDate = voiceWork.ExpectedDate, ExpectedDate = voiceWork.ExpectedDate,
SalesDate = voiceWork.SalesDate, SalesDate = voiceWork.SalesDate,
PlannedReleaseDate = voiceWork.PlannedReleaseDate, PlannedReleaseDate = voiceWork.PlannedReleaseDate,
@@ -413,11 +442,19 @@ public class VoiceWorkSearchProvider(AppDbContext context, IVoiceWorkFullTextSea
HasImage = voiceWork.HasImage, HasImage = voiceWork.HasImage,
Maker = circle.Name, Maker = circle.Name,
MakerId = circle.MakerId, MakerId = circle.MakerId,
Circle = new()
{
Name = circle.Name,
MakerId = circle.MakerId,
IsFavorite = circle.Favorite,
IsBlacklisted = circle.Blacklisted
},
ExpectedDate = voiceWork.ExpectedDate, ExpectedDate = voiceWork.ExpectedDate,
SalesDate = voiceWork.SalesDate, SalesDate = voiceWork.SalesDate,
PlannedReleaseDate = voiceWork.PlannedReleaseDate, PlannedReleaseDate = voiceWork.PlannedReleaseDate,
Downloads = voiceWork.Downloads, Downloads = voiceWork.Downloads,
WishlistCount = voiceWork.WishlistCount, WishlistCount = voiceWork.WishlistCount,
Rating = (AgeRating)voiceWork.Rating,
Status = voiceWork.Status, Status = voiceWork.Status,
SubtitleLanguage = voiceWork.SubtitleLanguage, SubtitleLanguage = voiceWork.SubtitleLanguage,
HasTrial = voiceWork.HasTrial, HasTrial = voiceWork.HasTrial,
@@ -435,16 +472,24 @@ public class VoiceWorkSearchProvider(AppDbContext context, IVoiceWorkFullTextSea
int[] voiceWorkIds = [.. items.Select(i => i.VoiceWorkId)]; int[] voiceWorkIds = [.. items.Select(i => i.VoiceWorkId)];
Dictionary<int, VoiceWorkTagItem[]> tagsByVw = await GetTagsAsync(voiceWorkIds, cancellationToken); var tagsByVoiceWork = await GetTagsAsync(voiceWorkIds, cancellationToken);
Dictionary<int, VoiceWorkCreatorItem[]> creatorsByVw = await GetCreatorsAsync(voiceWorkIds, cancellationToken); var creatorsByVoiceWork = await GetCreatorsAsync(voiceWorkIds, cancellationToken);
var supportedLanguagesByVoiceWork = await GetSupportedLanguagesAsync(voiceWorkIds, cancellationToken);
var originalCirclesByVoiceWork = await GetOriginalCircles(voiceWorkIds, cancellationToken);
foreach (VoiceWorkSearchResult item in items) foreach (VoiceWorkSearchResult item in items)
{ {
if (tagsByVw.TryGetValue(item.VoiceWorkId, out VoiceWorkTagItem[]? tags)) if (tagsByVoiceWork.TryGetValue(item.VoiceWorkId, out VoiceWorkTagItem[]? tags))
item.Tags = tags; item.Tags = tags;
if (creatorsByVw.TryGetValue(item.VoiceWorkId, out VoiceWorkCreatorItem[]? creators)) if (creatorsByVoiceWork.TryGetValue(item.VoiceWorkId, out VoiceWorkCreatorItem[]? creators))
item.Creators = creators; item.Creators = creators;
if (supportedLanguagesByVoiceWork.TryGetValue(item.VoiceWorkId, out string[]? supportedLanguages))
item.SupportedLanguages = supportedLanguages;
if (originalCirclesByVoiceWork.TryGetValue(item.VoiceWorkId, out VoiceWorkCircleItem? voiceWorkCircleItem))
item.OriginalCircle = voiceWorkCircleItem;
} }
} }
@@ -457,14 +502,14 @@ public class VoiceWorkSearchProvider(AppDbContext context, IVoiceWorkFullTextSea
from englishTag in et.DefaultIfEmpty() from englishTag in et.DefaultIfEmpty()
where voiceWorkIds.Contains(voiceWorkTag.VoiceWorkId) where voiceWorkIds.Contains(voiceWorkTag.VoiceWorkId)
orderby voiceWorkTag.VoiceWorkId, voiceWorkTag.Position orderby voiceWorkTag.VoiceWorkId, voiceWorkTag.Position
select new { voiceWorkTag.VoiceWorkId, voiceWorkTag.TagId, tag.Name, EnglishName = englishTag.Name } select new { voiceWorkTag.VoiceWorkId, voiceWorkTag.TagId, tag.Name, EnglishName = englishTag.Name, IsFavorite = tag.Favorite, IsBlacklisted = tag.Blacklisted }
).ToListAsync(cancellationToken); ).ToListAsync(cancellationToken);
return tagRows return tagRows
.GroupBy(r => r.VoiceWorkId) .GroupBy(r => r.VoiceWorkId)
.ToDictionary( .ToDictionary(
g => g.Key, g => g.Key,
g => g.Select(r => new VoiceWorkTagItem { TagId = r.TagId, Name = r.EnglishName ?? r.Name }).ToArray() g => g.Select(r => new VoiceWorkTagItem { TagId = r.TagId, Name = r.EnglishName ?? r.Name, IsFavorite = r.IsFavorite, IsBlacklisted = r.IsBlacklisted }).ToArray()
); );
} }
@@ -475,14 +520,61 @@ public class VoiceWorkSearchProvider(AppDbContext context, IVoiceWorkFullTextSea
join creator in context.Creators.AsNoTracking() on voiceWorkCreator.CreatorId equals creator.CreatorId join creator in context.Creators.AsNoTracking() on voiceWorkCreator.CreatorId equals creator.CreatorId
where voiceWorkIds.Contains(voiceWorkCreator.VoiceWorkId) where voiceWorkIds.Contains(voiceWorkCreator.VoiceWorkId)
orderby voiceWorkCreator.VoiceWorkId, voiceWorkCreator.Position orderby voiceWorkCreator.VoiceWorkId, voiceWorkCreator.Position
select new { voiceWorkCreator.VoiceWorkId, creator.CreatorId, creator.Name } select new { voiceWorkCreator.VoiceWorkId, creator.CreatorId, creator.Name, creator.Favorite, creator.Blacklisted }
).ToListAsync(cancellationToken); ).ToListAsync(cancellationToken);
return creatorRows return creatorRows
.GroupBy(r => r.VoiceWorkId) .GroupBy(r => r.VoiceWorkId)
.ToDictionary( .ToDictionary(
g => g.Key, g => g.Key,
g => g.Select(r => new VoiceWorkCreatorItem { CreatorId = r.CreatorId, Name = r.Name }).ToArray() g => g.Select(r => new VoiceWorkCreatorItem { CreatorId = r.CreatorId, Name = r.Name, IsFavorite = r.Favorite, IsBlacklisted = r.Blacklisted }).ToArray()
);
}
private async Task<Dictionary<int, string[]>> GetSupportedLanguagesAsync(int[] voiceWorkIds, CancellationToken cancellationToken)
{
var supportedLanguageRows = await (
from voiceWorkSupportedLanguage in context.VoiceWorkSupportedLanguages.AsNoTracking()
where voiceWorkIds.Contains(voiceWorkSupportedLanguage.VoiceWorkId)
select new { voiceWorkSupportedLanguage.VoiceWorkId, voiceWorkSupportedLanguage.Language }
).ToListAsync(cancellationToken);
return supportedLanguageRows
.GroupBy(r => r.VoiceWorkId)
.ToDictionary(
g => g.Key,
g => g.Select(r => r.Language).ToArray()
);
}
private async Task<Dictionary<int, VoiceWorkCircleItem>> GetOriginalCircles(int[] voiceWorkIds, CancellationToken cancellationToken)
{
var originalCircleRows = await (
from voiceWork in context.VoiceWorks.AsNoTracking()
join orignalVoiceWork in context.VoiceWorks.AsNoTracking() on voiceWork.OriginalProductId equals orignalVoiceWork.ProductId
join originalCircle in context.Circles.AsNoTracking() on orignalVoiceWork.CircleId equals originalCircle.CircleId
where voiceWorkIds.Contains(voiceWork.VoiceWorkId)
select new
{
voiceWork.VoiceWorkId,
originalCircle.Name,
originalCircle.MakerId,
originalCircle.Favorite,
originalCircle.Blacklisted
}
).ToListAsync(cancellationToken);
return originalCircleRows
.GroupBy(r => r.VoiceWorkId)
.ToDictionary(
g => g.Key,
g => g.Select(r => new VoiceWorkCircleItem()
{
Name = r.Name,
MakerId = r.MakerId,
IsFavorite = r.Favorite,
IsBlacklisted = r.Blacklisted
}).First()
); );
} }
} }

View File

@@ -1,7 +1,7 @@
using JSMR.Application.VoiceWorks.Commands.SetFavorite; using JSMR.Application.VoiceWorks.Commands.Delete;
using JSMR.Application.VoiceWorks.Commands.SetFavorite;
using JSMR.Application.VoiceWorks.Ports; using JSMR.Application.VoiceWorks.Ports;
using JSMR.Domain.Entities; using JSMR.Domain.Entities;
using JSMR.Infrastructure.Common.Time;
using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore;
namespace JSMR.Infrastructure.Data.Repositories.VoiceWorks; namespace JSMR.Infrastructure.Data.Repositories.VoiceWorks;
@@ -18,6 +18,32 @@ public class VoiceWorkWriter(AppDbContext dbContext) : IVoiceWorkWriter
return new SetVoiceWorkFavoriteResponse(request.VoiceWorkId, request.IsFavorite); return new SetVoiceWorkFavoriteResponse(request.VoiceWorkId, request.IsFavorite);
} }
public async Task<DeleteVoiceWorkResponse> DeleteAsync(DeleteVoiceWorkRequest request, CancellationToken cancellationToken)
{
Dictionary<int, bool> isSuccess = [];
VoiceWork[] voiceWorks = [.. dbContext.VoiceWorks.Where(voiceWork => request.VoiceWorkIds.Contains(voiceWork.VoiceWorkId))
.Include(x => x.Circle)];
foreach (VoiceWork voiceWork in voiceWorks)
{
isSuccess.Add(voiceWork.VoiceWorkId, false);
if (voiceWork.Circle is null)
continue;
if (voiceWork.IsValid == true && voiceWork.Circle.Spam == false)
continue;
dbContext.Remove(voiceWork);
isSuccess[voiceWork.VoiceWorkId] = true;
}
dbContext.SaveChanges();
return new DeleteVoiceWorkResponse(isSuccess);
}
private async Task<VoiceWork> GetVoiceWorkAsync(int voiceWorkId, CancellationToken cancellationToken) private async Task<VoiceWork> GetVoiceWorkAsync(int voiceWorkId, CancellationToken cancellationToken)
{ {
return await dbContext.VoiceWorks.FirstOrDefaultAsync(voiceWork => voiceWork.VoiceWorkId == voiceWorkId, cancellationToken) return await dbContext.VoiceWorks.FirstOrDefaultAsync(voiceWork => voiceWork.VoiceWorkId == voiceWorkId, cancellationToken)

View File

@@ -0,0 +1,48 @@
using JSMR.Application.Enums;
namespace JSMR.Infrastructure.Globalization;
internal class LocaleMapper
{
// TODO: Deprecate
public static readonly IReadOnlyDictionary<Locale, (string Abbreviation, string Code)> Map =
new Dictionary<Locale, (string, string)>
{
{ Locale.Japanese, ("jp", "ja_JP") },
{ Locale.English, ("en", "en_US") },
{ Locale.ChineseSimplified, ("zh-cn", "zh_CN") },
{ Locale.ChineseTraditional, ("zh-tw", "zh_TW") },
{ Locale.Korean, ("ko", "ko_KR") },
};
public static string ToDLSiteLocale(Locale locale) => locale switch
{
Locale.Japanese => "ja-jp",
Locale.English => "en-us",
Locale.ChineseSimplified => "zh-cn",
Locale.ChineseTraditional => "zh-tw",
Locale.Korean => "ko-kr",
_ => throw new NotSupportedException($"Locale '{locale}' is not supported.")
};
public static string ToDLSiteApiLocale(Locale locale) => locale switch
{
Locale.Japanese => "ja_JP",
Locale.English => "en_US",
Locale.ChineseSimplified => "zh_CN",
Locale.ChineseTraditional => "zh_TW",
Locale.Korean => "ko_KR",
_ => throw new NotSupportedException($"Locale '{locale}' is not supported.")
};
public static string ToAbbreviation(Locale locale) => locale switch
{
Locale.Japanese => "jp",
Locale.English => "en",
Locale.ChineseSimplified => "zh-cn",
Locale.ChineseTraditional => "zh-tw",
Locale.Korean => "ko",
_ => throw new NotSupportedException($"Locale '{locale}' is not supported.")
};
}

View File

@@ -1,88 +1,134 @@
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
using System.IO;
using System.Net.Http.Headers; using System.Net.Http.Headers;
using System.Text; using System.Text;
using System.Text.Json; using System.Text.Json;
namespace JSMR.Infrastructure.Http; namespace JSMR.Infrastructure.Http;
public abstract class ApiClient(IHttpService http, ILogger logger, JsonSerializerOptions? json = null) public abstract class ApiClient(HttpClient http, ILogger logger, JsonSerializerOptions? json = null)
{ {
protected async Task<T> GetJsonAsync<T>(string url, CancellationToken cancellationToken = default) protected async Task<TResponse> GetJsonAsync<TResponse>(
string url,
Action<HttpRequestHeaders>? configureHeaders = null,
CancellationToken cancellationToken = default)
{ {
string response = await http.GetStringAsync(url, cancellationToken); using HttpRequestMessage request = new(HttpMethod.Get, url);
configureHeaders?.Invoke(request.Headers);
return JsonSerializer.Deserialize<T>(response, json) LogRequest(request);
?? throw new InvalidOperationException($"Failed to deserialize JSON to {typeof(T).Name} from {url}.");
using HttpResponseMessage response = await http.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
await EnsureSuccess(response).ConfigureAwait(false);
Stream stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
try
{
return await JsonSerializer.DeserializeAsync<TResponse>(stream, json, cancellationToken).ConfigureAwait(false)
?? throw new InvalidOperationException($"Failed to deserialize JSON to {typeof(TResponse).Name} from {url}.");
}
catch (JsonException ex)
{
logger.LogError(ex,
"Failed to deserialize JSON from {Url}. ContentLengthHeader={ContentLengthHeader}",
url,
response.Content.Headers.ContentLength);
throw;
} }
//protected async Task<T> GetJsonAsync<T>( }
// string url,
// Action<HttpRequestHeaders>? configureHeaders = null,
// CancellationToken ct = default
// )
//{
// using var req = new HttpRequestMessage(HttpMethod.Get, url);
// configureHeaders?.Invoke(req.Headers);
// LogRequest(req); protected async Task<TResponse> GetJsonpAsync<TResponse>(
string url,
Action<HttpRequestHeaders>? configureHeaders = null,
CancellationToken cancellationToken = default)
{
using HttpRequestMessage request = new(HttpMethod.Get, url);
configureHeaders?.Invoke(request.Headers);
// using var res = await http.SendAsync(req, HttpCompletionOption.ResponseHeadersRead, ct).ConfigureAwait(false); LogRequest(request);
// await EnsureSuccess(res).ConfigureAwait(false);
// var stream = await res.Content.ReadAsStreamAsync(ct).ConfigureAwait(false); using HttpResponseMessage response = await http.SendAsync(
request,
HttpCompletionOption.ResponseHeadersRead,
cancellationToken).ConfigureAwait(false);
// var model = await JsonSerializer.DeserializeAsync<T>(stream, json, ct).ConfigureAwait(false) await EnsureSuccess(response).ConfigureAwait(false);
// ?? throw new InvalidOperationException($"Failed to deserialize JSON to {typeof(T).Name} from {url}.");
// return model; string body = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
//}
//protected async Task<TResponse> PostJsonAsync<TRequest, TResponse>( string jsonBody = ExtractJsonFromJsonp(body);
// string url,
// TRequest payload,
// Action<HttpRequestHeaders>? configureHeaders = null,
// CancellationToken ct = default)
//{
// var content = new StringContent(JsonSerializer.Serialize(payload, json), Encoding.UTF8, "application/json");
// using var req = new HttpRequestMessage(HttpMethod.Post, url) { Content = content };
// configureHeaders?.Invoke(req.Headers);
// LogRequest(req); return JsonSerializer.Deserialize<TResponse>(jsonBody, json)
?? throw new InvalidOperationException($"Failed to deserialize JSONP payload to {typeof(TResponse).Name} from {url}.");
}
// using var res = await http.SendAsync(req, HttpCompletionOption.ResponseHeadersRead, ct).ConfigureAwait(false); private static string ExtractJsonFromJsonp(string body)
// await EnsureSuccess(res).ConfigureAwait(false); {
if (string.IsNullOrWhiteSpace(body))
throw new InvalidOperationException("Response body was empty.");
// var stream = await res.Content.ReadAsStreamAsync(ct).ConfigureAwait(false); body = body.Trim();
// var model = await JsonSerializer.DeserializeAsync<TResponse>(stream, json, ct).ConfigureAwait(false) int firstParen = body.IndexOf('(');
// ?? throw new InvalidOperationException($"Failed to deserialize JSON to {typeof(TResponse).Name} from {url}."); int lastParen = body.LastIndexOf(')');
// return model; if (firstParen < 0 || lastParen <= firstParen)
//} throw new InvalidOperationException("Response was not valid JSONP.");
//protected virtual void LogRequest(HttpRequestMessage req) return body[(firstParen + 1)..lastParen].Trim();
// => logger.LogDebug("HTTP {Method} {Uri}", req.Method, req.RequestUri); }
//protected virtual void LogFailure(HttpResponseMessage res, string body) protected async Task<TResponse> PostJsonAsync<TRequest, TResponse>(
// => logger.LogWarning("HTTP {Status} for {Uri}. Body: {Body}", (int)res.StatusCode, res.RequestMessage?.RequestUri, Truncate(body, 500)); string url,
TRequest payload,
Action<HttpRequestHeaders>? configureHeaders = null,
CancellationToken cancellationToken = default)
{
StringContent content = new(JsonSerializer.Serialize(payload, json), Encoding.UTF8, "application/json");
//protected static string Truncate(string s, int max) => s.Length <= max ? s : s[..max] + "…"; using HttpRequestMessage request = new(HttpMethod.Post, url)
{
Content = content
};
//protected async Task EnsureSuccess(HttpResponseMessage res) configureHeaders?.Invoke(request.Headers);
//{
// if (res.IsSuccessStatusCode) return;
// string body; LogRequest(request);
// try { body = await res.Content.ReadAsStringAsync().ConfigureAwait(false); }
// catch { body = "<unable to read body>"; }
// LogFailure(res, body); using HttpResponseMessage response = await http.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
await EnsureSuccess(response).ConfigureAwait(false);
// Throw a richer exception(you can customize per API) Stream stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
// throw new HttpRequestException(
// $"Request to {res.RequestMessage?.RequestUri} failed: {(int)res.StatusCode} {res.ReasonPhrase}. Body: {Truncate(body, 1000)}", return await JsonSerializer.DeserializeAsync<TResponse>(stream, json, cancellationToken).ConfigureAwait(false)
// null, ?? throw new InvalidOperationException($"Failed to deserialize JSON to {typeof(TResponse).Name} from {url}.");
// res.StatusCode); }
//}
protected virtual void LogRequest(HttpRequestMessage request)
=> logger.LogDebug("HTTP {Method} {Uri}", request.Method, request.RequestUri);
protected virtual void LogFailure(HttpResponseMessage response, string body)
=> logger.LogWarning("HTTP {Status} for {Uri}. Body: {Body}", (int)response.StatusCode, response.RequestMessage?.RequestUri, Truncate(body, 500));
protected static string Truncate(string s, int max) => s.Length <= max ? s : s[..max] + "…";
protected async Task EnsureSuccess(HttpResponseMessage res)
{
if (res.IsSuccessStatusCode)
return;
string body;
try { body = await res.Content.ReadAsStringAsync().ConfigureAwait(false); }
catch { body = "<unable to read body>"; }
LogFailure(res, body);
//Throw a richer exception(you can customize per API)
throw new HttpRequestException(
$"Request to {res.RequestMessage?.RequestUri} failed: {(int)res.StatusCode} {res.ReasonPhrase}. Body: {Truncate(body, 1000)}",
null,
res.StatusCode);
}
} }

View File

@@ -0,0 +1,12 @@
using HtmlAgilityPack;
using System.Net;
namespace JSMR.Infrastructure.Http;
public sealed class HtmlLoadResult
{
public required HttpStatusCode StatusCode { get; init; }
public HtmlDocument? Document { get; init; }
public bool IsSuccessStatusCode => (int)StatusCode is >= 200 and <= 299;
}

View File

@@ -4,13 +4,26 @@ namespace JSMR.Infrastructure.Http;
public class HtmlLoader(IHttpService httpService) : IHtmlLoader public class HtmlLoader(IHttpService httpService) : IHtmlLoader
{ {
public async Task<HtmlDocument> GetHtmlDocumentAsync(string url, CancellationToken cancellationToken) public async Task<HtmlLoadResult> GetHtmlDocumentAsync(string url, CancellationToken cancellationToken)
{ {
string html = await httpService.GetStringAsync(url, cancellationToken); HttpStringResponse response = await httpService.GetAsync(url, cancellationToken);
if (!response.IsSuccessStatusCode)
{
return new HtmlLoadResult
{
StatusCode = response.StatusCode,
Document = null
};
}
HtmlDocument document = new(); HtmlDocument document = new();
document.LoadHtml(html); document.LoadHtml(response.Content ?? string.Empty);
return document; return new HtmlLoadResult
{
StatusCode = response.StatusCode,
Document = document
};
} }
} }

View File

@@ -2,10 +2,10 @@
public class HttpService(HttpClient httpClient) : IHttpService public class HttpService(HttpClient httpClient) : IHttpService
{ {
public Task<string> GetStringAsync(string url, CancellationToken cancellationToken) public Task<HttpStringResponse> GetAsync(string url, CancellationToken cancellationToken)
=> GetStringAsync(url, new Dictionary<string, string>(), cancellationToken); => GetAsync(url, new Dictionary<string, string>(), cancellationToken);
public async Task<string> GetStringAsync(string url, IDictionary<string, string> headers, CancellationToken cancellationToken) public async Task<HttpStringResponse> GetAsync(string url, IDictionary<string, string> headers, CancellationToken cancellationToken)
{ {
using HttpRequestMessage request = new(HttpMethod.Get, url); using HttpRequestMessage request = new(HttpMethod.Get, url);
@@ -14,11 +14,18 @@ public class HttpService(HttpClient httpClient) : IHttpService
request.Headers.TryAddWithoutValidation(header.Key, header.Value); request.Headers.TryAddWithoutValidation(header.Key, header.Value);
} }
httpClient.DefaultRequestHeaders.UserAgent.ParseAdd("JSMR/1.0"); request.Headers.UserAgent.ParseAdd("JSMR/1.0");
using HttpResponseMessage response = await httpClient.SendAsync(request, cancellationToken); using HttpResponseMessage response = await httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken);
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync(cancellationToken); string? content = response.Content is null
? null
: await response.Content.ReadAsStringAsync(cancellationToken);
return new HttpStringResponse
{
StatusCode = response.StatusCode,
Content = content
};
} }
} }

View File

@@ -0,0 +1,11 @@
using System.Net;
namespace JSMR.Infrastructure.Http;
public sealed class HttpStringResponse
{
public required HttpStatusCode StatusCode { get; init; }
public string? Content { get; init; }
public bool IsSuccessStatusCode => (int)StatusCode is >= 200 and <= 299;
}

View File

@@ -1,8 +1,6 @@
using HtmlAgilityPack; namespace JSMR.Infrastructure.Http;
namespace JSMR.Infrastructure.Http;
public interface IHtmlLoader public interface IHtmlLoader
{ {
Task<HtmlDocument> GetHtmlDocumentAsync(string url, CancellationToken cancellationToken); Task<HtmlLoadResult> GetHtmlDocumentAsync(string url, CancellationToken cancellationToken);
} }

View File

@@ -2,6 +2,6 @@
public interface IHttpService public interface IHttpService
{ {
Task<string> GetStringAsync(string url, CancellationToken cancellationToken); Task<HttpStringResponse> GetAsync(string url, CancellationToken cancellationToken);
Task<string> GetStringAsync(string url, IDictionary<string, string> headers, CancellationToken cancellationToken); Task<HttpStringResponse> GetAsync(string url, IDictionary<string, string> headers, CancellationToken cancellationToken);
} }

View File

@@ -56,6 +56,10 @@ public class EnglishVoiceWorkUpdater(AppDbContext dbContext, ILanguageIdentifier
Results: productIds.ToDictionary( Results: productIds.ToDictionary(
productId => productId, productId => productId,
productId => new VoiceWorkUpsertResult() productId => new VoiceWorkUpsertResult()
{
ProductId = productId,
Title = ingests.First(x => x.ProductId == productId).Title
}
) )
); );

View File

@@ -11,20 +11,31 @@ public class VoiceWorkSearchUpdater(AppDbContext dbContext) : IVoiceWorkSearchUp
public async Task UpdateAsync(int[] voiceWorkIds, CancellationToken cancellationToken) public async Task UpdateAsync(int[] voiceWorkIds, CancellationToken cancellationToken)
{ {
List<VoiceWork> batch = await dbContext.VoiceWorks List<VoiceWork> batch = await dbContext.VoiceWorks
.Where(vw => voiceWorkIds.Contains(vw.VoiceWorkId))
.AsSplitQuery()
.Include(vw => vw.Circle) .Include(vw => vw.Circle)
.Include(vw => vw.Tags) .Include(vw => vw.Tags)
.ThenInclude(vwt => vwt.Tag) .ThenInclude(vwt => vwt.Tag)
.Include(vw => vw.Creators) .Include(vw => vw.Creators)
.ThenInclude(vwc => vwc.Creator) .ThenInclude(vwc => vwc.Creator)
.Include(vw => vw.EnglishVoiceWorks) .Include(vw => vw.EnglishVoiceWorks)
.Where(vw => voiceWorkIds.Contains(vw.VoiceWorkId))
.ToListAsync(cancellationToken); .ToListAsync(cancellationToken);
Dictionary<int, VoiceWorkSearch> existingSearches = await dbContext.VoiceWorkSearches
.Where(s => voiceWorkIds.Contains(s.VoiceWorkId))
.ToDictionaryAsync(s => s.VoiceWorkId, cancellationToken);
int[] tagIds = [.. batch.SelectMany(vw => vw.Tags).Select(vwt => vwt.TagId).Distinct()];
Dictionary<int, EnglishTag> englishTags = await dbContext.EnglishTags
.Where(et => tagIds.Contains(et.TagId))
.ToDictionaryAsync(et => et.TagId, cancellationToken);
foreach (var voiceWork in batch) foreach (var voiceWork in batch)
{ {
try try
{ {
UpdateSearchText(voiceWork); UpdateSearchText(voiceWork, existingSearches, englishTags);
} }
catch (Exception ex) catch (Exception ex)
{ {
@@ -32,17 +43,14 @@ public class VoiceWorkSearchUpdater(AppDbContext dbContext) : IVoiceWorkSearchUp
} }
} }
dbContext.SaveChanges(); await dbContext.SaveChangesAsync(cancellationToken);
} }
private void UpdateSearchText(VoiceWork voiceWork) private void UpdateSearchText(VoiceWork voiceWork, Dictionary<int, VoiceWorkSearch> existingSearches, Dictionary<int, EnglishTag> englishTags)
{ {
string searchText = GetSearchText(voiceWork); string searchText = GetSearchText(voiceWork, englishTags);
var searchEntry = dbContext.VoiceWorkSearches if (!existingSearches.TryGetValue(voiceWork.VoiceWorkId, out var searchEntry))
.FirstOrDefault(s => s.VoiceWorkId == voiceWork.VoiceWorkId);
if (searchEntry == null)
{ {
dbContext.VoiceWorkSearches.Add(new VoiceWorkSearch dbContext.VoiceWorkSearches.Add(new VoiceWorkSearch
{ {
@@ -51,12 +59,15 @@ public class VoiceWorkSearchUpdater(AppDbContext dbContext) : IVoiceWorkSearchUp
}); });
} }
else else
{
if (!string.Equals(searchEntry.SearchText, searchText, StringComparison.Ordinal))
{ {
searchEntry.SearchText = searchText; searchEntry.SearchText = searchText;
} }
} }
}
private string GetSearchText(VoiceWork voiceWork) private string GetSearchText(VoiceWork voiceWork, Dictionary<int, EnglishTag> englishTags)
{ {
var english = voiceWork.EnglishVoiceWorks.FirstOrDefault(); var english = voiceWork.EnglishVoiceWorks.FirstOrDefault();
@@ -79,12 +90,8 @@ public class VoiceWorkSearchUpdater(AppDbContext dbContext) : IVoiceWorkSearchUp
AppendRaw(sb, tag.Name); AppendRaw(sb, tag.Name);
var englishTag = dbContext.EnglishTags.FirstOrDefault(et => et.TagId == tag.TagId); if (englishTags.TryGetValue(tag.TagId, out var englishTag))
AppendRaw(sb, englishTag.Name);
if (englishTag is null)
continue;
AppendRaw(sb, englishTag?.Name);
} }
foreach (var creator in voiceWork.Creators.Select(vwc => vwc.Creator)) foreach (var creator in voiceWork.Creators.Select(vwc => vwc.Creator))

View File

@@ -28,10 +28,25 @@ public class VoiceWorkUpdater(AppDbContext dbContext, ITimeProvider timeProvider
} }
result.Status = Upsert(ingest, upsertContext); result.Status = Upsert(ingest, upsertContext);
VoiceWork voiceWork = upsertContext.VoiceWorks[ingest.ProductId];
result.UpdateStatus = (VoiceWorkStatus)voiceWork.Status;
} }
await dbContext.SaveChangesAsync(cancellationToken); await dbContext.SaveChangesAsync(cancellationToken);
foreach (VoiceWorkIngest ingest in ingests)
{
VoiceWorkUpsertResult result = upsertContext.Results[ingest.ProductId];
if (result.Status is VoiceWorkUpsertStatus.Skipped)
continue;
VoiceWork voiceWork = upsertContext.VoiceWorks[ingest.ProductId];
result.VoiceWorkId = voiceWork.VoiceWorkId;
}
return [.. upsertContext.Results.Select(x => x.Value)]; return [.. upsertContext.Results.Select(x => x.Value)];
} }
@@ -54,8 +69,10 @@ public class VoiceWorkUpdater(AppDbContext dbContext, ITimeProvider timeProvider
.ToDictionaryAsync(c => c.MakerId, cancellationToken), .ToDictionaryAsync(c => c.MakerId, cancellationToken),
VoiceWorks: await dbContext.VoiceWorks VoiceWorks: await dbContext.VoiceWorks
.Where(v => productIds.Contains(v.ProductId)) .Where(v => productIds.Contains(v.ProductId))
.AsSplitQuery()
.Include(v => v.Creators) .Include(v => v.Creators)
.Include(v => v.Tags) .Include(v => v.Tags)
.Include(v => v.EnglishVoiceWorks)
.Include(v => v.Localizations) .Include(v => v.Localizations)
.Include(v => v.SupportedLanguages) .Include(v => v.SupportedLanguages)
.ToDictionaryAsync(v => v.ProductId, cancellationToken), .ToDictionaryAsync(v => v.ProductId, cancellationToken),
@@ -71,6 +88,10 @@ public class VoiceWorkUpdater(AppDbContext dbContext, ITimeProvider timeProvider
Results: productIds.ToDictionary( Results: productIds.ToDictionary(
productId => productId, productId => productId,
productId => new VoiceWorkUpsertResult() productId => new VoiceWorkUpsertResult()
{
ProductId = productId,
Title = ingests.First(x => x.ProductId == productId).Title
}
) )
); );
@@ -124,7 +145,7 @@ public class VoiceWorkUpdater(AppDbContext dbContext, ITimeProvider timeProvider
if (voiceWork.SalesDate is not null && ingest.SalesDate is null) if (voiceWork.SalesDate is not null && ingest.SalesDate is null)
{ {
string message = $"Voice work has a sales date of {voiceWork.SalesDate.Value.ToShortDateString()}, but ingest does not"; string message = $"Voice work has a sales date of {voiceWork.SalesDate.Value:d}, but ingest does not";
result.Issues.Add(new(message, VoiceWorkUpsertIssueSeverity.Error)); result.Issues.Add(new(message, VoiceWorkUpsertIssueSeverity.Error));
} }
} }
@@ -141,6 +162,7 @@ public class VoiceWorkUpdater(AppDbContext dbContext, ITimeProvider timeProvider
UpsertVoiceWorkCreators(ingest, upsertContext); UpsertVoiceWorkCreators(ingest, upsertContext);
UpsertVoiceWorkSupportedLanguages(ingest, upsertContext); UpsertVoiceWorkSupportedLanguages(ingest, upsertContext);
UpsertSeries(ingest, upsertContext); UpsertSeries(ingest, upsertContext);
UpsertVoiceWorkLocalizations(ingest, upsertContext);
return dbContext.Entry(voiceWork).State switch return dbContext.Entry(voiceWork).State switch
{ {
@@ -186,10 +208,11 @@ public class VoiceWorkUpdater(AppDbContext dbContext, ITimeProvider timeProvider
voiceWork.Downloads = ingest.Downloads; voiceWork.Downloads = ingest.Downloads;
voiceWork.WishlistCount = ingest.WishlistCount; voiceWork.WishlistCount = ingest.WishlistCount;
voiceWork.HasTrial = ingest.HasTrial; voiceWork.HasTrial = ingest.HasTrial;
voiceWork.HasChobit = ingest.HasDLPlay; voiceWork.HasChobit = ingest.HasChobit;
voiceWork.StarRating = ingest.StarRating; voiceWork.StarRating = ingest.StarRating;
voiceWork.Votes = ingest.Votes; voiceWork.Votes = ingest.Votes;
voiceWork.OriginalProductId = ingest.Translation?.OriginalProductId; voiceWork.OriginalProductId = ingest.Translation?.OriginalProductId;
voiceWork.SubtitleLanguage = GetSubtitleLanguage(ingest);
voiceWork.AIGeneration = (byte)ingest.AI; voiceWork.AIGeneration = (byte)ingest.AI;
voiceWork.IsValid = true; voiceWork.IsValid = true;
voiceWork.LastScannedDate = ComputeLastScannedDate(voiceWork.LastScannedDate, state, upsertContext); voiceWork.LastScannedDate = ComputeLastScannedDate(voiceWork.LastScannedDate, state, upsertContext);
@@ -204,7 +227,7 @@ public class VoiceWorkUpdater(AppDbContext dbContext, ITimeProvider timeProvider
else else
{ {
voiceWork.SalesDate = null; voiceWork.SalesDate = null;
voiceWork.ExpectedDate = ingest.ExpectedDate?.ToDateTime(new TimeOnly(0, 0)); voiceWork.ExpectedDate = ingest.ExpectedDate?.ToDateTime(new TimeOnly(0, 0)) ?? voiceWork.ExpectedDate;
voiceWork.PlannedReleaseDate = ingest.RegistrationDate > upsertContext.CurrentScanAnchor ? ingest.RegistrationDate : null; voiceWork.PlannedReleaseDate = ingest.RegistrationDate > upsertContext.CurrentScanAnchor ? ingest.RegistrationDate : null;
voiceWork.Status = state.IsNewUpcoming ? (byte)VoiceWorkStatus.NewAndUpcoming : (byte)VoiceWorkStatus.Upcoming; voiceWork.Status = state.IsNewUpcoming ? (byte)VoiceWorkStatus.NewAndUpcoming : (byte)VoiceWorkStatus.Upcoming;
} }
@@ -266,6 +289,36 @@ public class VoiceWorkUpdater(AppDbContext dbContext, ITimeProvider timeProvider
return state.WentOnSale ? current : existing ?? current; return state.WentOnSale ? current : existing ?? current;
} }
private static byte GetSubtitleLanguage(VoiceWorkIngest ingest)
{
Language[] orderedLanguages =
[
Language.English,
Language.ChineseSimplified,
Language.ChineseTraditional,
Language.Korean
];
foreach (Language language in orderedLanguages)
{
if (ingest.SupportedLanguages.Any(x => x.Language == language))
{
switch (language)
{
case Language.English:
return 1;
case Language.ChineseSimplified:
case Language.ChineseTraditional:
return 2;
case Language.Korean:
return 3;
}
}
}
return 0;
}
private void UpsertTags(VoiceWorkIngest ingest, VoiceWorkUpsertContext upsertContext) private void UpsertTags(VoiceWorkIngest ingest, VoiceWorkUpsertContext upsertContext)
{ {
foreach (string tagName in ingest.Tags) foreach (string tagName in ingest.Tags)
@@ -321,7 +374,9 @@ public class VoiceWorkUpdater(AppDbContext dbContext, ITimeProvider timeProvider
int position = 1; int position = 1;
foreach (string tagName in ingest.Tags) string[] distinctTagNames = [.. ingest.Tags.Distinct()];
foreach (string tagName in distinctTagNames)
{ {
Tag tag = upsertContext.Tags[tagName]; Tag tag = upsertContext.Tags[tagName];
@@ -348,7 +403,9 @@ public class VoiceWorkUpdater(AppDbContext dbContext, ITimeProvider timeProvider
int position = 1; int position = 1;
foreach (string creatorName in ingest.Creators) string[] distinctCreatorNames = [.. ingest.Creators.Distinct()];
foreach (string creatorName in distinctCreatorNames)
{ {
Creator creator = upsertContext.Creators[creatorName]; Creator creator = upsertContext.Creators[creatorName];
@@ -373,6 +430,9 @@ public class VoiceWorkUpdater(AppDbContext dbContext, ITimeProvider timeProvider
VoiceWork voiceWork = upsertContext.VoiceWorks[ingest.ProductId]; VoiceWork voiceWork = upsertContext.VoiceWorks[ingest.ProductId];
Dictionary<string, VoiceWorkSupportedLanguage> existingLanguageLinks = voiceWork.SupportedLanguages.ToDictionary(x => x.Language); Dictionary<string, VoiceWorkSupportedLanguage> existingLanguageLinks = voiceWork.SupportedLanguages.ToDictionary(x => x.Language);
if (ingest.SupportedLanguages.Count == 0)
return;
foreach (SupportedLanguage supportedLanguage in ingest.SupportedLanguages) foreach (SupportedLanguage supportedLanguage in ingest.SupportedLanguages)
{ {
if (!existingLanguageLinks.TryGetValue(supportedLanguage.Code, out VoiceWorkSupportedLanguage? voiceWorkSupportedLanguage)) if (!existingLanguageLinks.TryGetValue(supportedLanguage.Code, out VoiceWorkSupportedLanguage? voiceWorkSupportedLanguage))
@@ -386,6 +446,14 @@ public class VoiceWorkUpdater(AppDbContext dbContext, ITimeProvider timeProvider
dbContext.VoiceWorkSupportedLanguages.Add(voiceWorkSupportedLanguage); dbContext.VoiceWorkSupportedLanguages.Add(voiceWorkSupportedLanguage);
} }
} }
foreach (string existingLinkCode in existingLanguageLinks.Keys)
{
if (!ingest.SupportedLanguages.Any(x => x.Code == existingLinkCode))
{
dbContext.VoiceWorkSupportedLanguages.Remove(existingLanguageLinks[existingLinkCode]);
}
}
} }
private void UpsertSeries(VoiceWorkIngest ingest, VoiceWorkUpsertContext upsertContext) private void UpsertSeries(VoiceWorkIngest ingest, VoiceWorkUpsertContext upsertContext)
@@ -422,4 +490,39 @@ public class VoiceWorkUpdater(AppDbContext dbContext, ITimeProvider timeProvider
return series; return series;
} }
private void UpsertVoiceWorkLocalizations(VoiceWorkIngest ingest, VoiceWorkUpsertContext upsertContext)
{
// For now, just adding/updating English voice works
foreach (VoiceWorkLocalizationIngest localizationIngest in ingest.Localizations)
{
if (localizationIngest.Language is Language.English)
{
EnglishVoiceWork englishVoiceWork = GetOrAddEnglishVoiceWork(ingest, upsertContext);
englishVoiceWork.ProductName = localizationIngest.Title ?? string.Empty;
englishVoiceWork.Description = localizationIngest.Description ?? string.Empty;
englishVoiceWork.IsValid = true;
}
}
}
private EnglishVoiceWork GetOrAddEnglishVoiceWork(VoiceWorkIngest ingest, VoiceWorkUpsertContext upsertContext)
{
VoiceWork voiceWork = upsertContext.VoiceWorks[ingest.ProductId];
EnglishVoiceWork? englishVoiceWork = voiceWork.EnglishVoiceWorks.FirstOrDefault();
if (englishVoiceWork is null)
{
englishVoiceWork = new EnglishVoiceWork
{
VoiceWork = voiceWork,
ProductName = string.Empty,
Description = string.Empty
};
dbContext.EnglishVoiceWorks.Add(englishVoiceWork);
}
return englishVoiceWork;
}
} }

View File

@@ -0,0 +1,13 @@
using JSMR.Application.Enums;
using JSMR.Application.Scanning.Ports;
using Microsoft.Extensions.DependencyInjection;
namespace JSMR.Infrastructure.Ingestion;
public class VoiceWorkUpdaterRepository(IServiceProvider serviceProvider) : IVoiceWorkUpdaterRepository
{
public IVoiceWorkUpdater? GetUpdater(Locale locale)
{
return serviceProvider.GetKeyedService<IVoiceWorkUpdater>(locale);
}
}

View File

@@ -1,15 +1,21 @@
using JSMR.Application.Integrations.Chobit.Models; using JSMR.Application.Integrations.Chobit.Models;
using JSMR.Application.Integrations.Ports; using JSMR.Application.Integrations.Chobit.Ports;
using JSMR.Infrastructure.Http; using JSMR.Infrastructure.Http;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
namespace JSMR.Infrastructure.Integrations.Chobit; namespace JSMR.Infrastructure.Integrations.Chobit;
public class ChobitClient(IHttpService http, ILogger logger) : ApiClient(http, logger), IChobitClient public class ChobitClient(HttpClient http, ILogger<ChobitClient> logger) : ApiClient(http, logger), IChobitClient
{ {
public Task<ChobitWorkResult> GetSampleInfoAsync(string[] productIds, CancellationToken cancellationToken = default) public Task<ChobitResult> GetSampleInfoAsync(string productId, CancellationToken cancellationToken = default)
{ {
var url = $"api/v2/dlsite/embed?workno_list=${string.Join(",", productIds)}"; var url = $"api/v1/dlsite/embed?workno={productId}";
return GetJsonAsync<ChobitWorkResult>(url, cancellationToken); return GetJsonpAsync<ChobitResult>(url, cancellationToken: cancellationToken);
}
public Task<ChobitResultCollection> GetSampleInfoAsync(string[] productIds, CancellationToken cancellationToken = default)
{
var url = $"api/v2/dlsite/embed?workno_list={string.Join(",", productIds)}";
return GetJsonpAsync<ChobitResultCollection>(url, cancellationToken: cancellationToken);
} }
} }

View File

@@ -1,28 +1,48 @@
using JSMR.Application.Integrations.DLSite.Models; using JSMR.Application.Integrations.DLSite.Models;
using JSMR.Application.Integrations.Ports; using JSMR.Application.Integrations.DLSite.Models.ReleasedWorks;
using JSMR.Application.Integrations.DLSite.Ports;
using JSMR.Infrastructure.Globalization;
using JSMR.Infrastructure.Http; using JSMR.Infrastructure.Http;
using JSMR.Infrastructure.Integrations.DLSite.Mapping; using JSMR.Infrastructure.Integrations.DLSite.Mapping;
using JSMR.Infrastructure.Integrations.DLSite.Models; using JSMR.Infrastructure.Integrations.DLSite.Models;
using JSMR.Infrastructure.Integrations.DLSite.Models.NewWorks;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
namespace JSMR.Infrastructure.Integrations.DLSite; namespace JSMR.Infrastructure.Integrations.DLSite;
public class DLSiteClient(IHttpService http, ILogger logger) : ApiClient(http, logger), IDLSiteClient public class DLSiteClient(HttpClient http, ILogger<DLSiteClient> logger) : ApiClient(http, logger), IDLSiteClient
{ {
public async Task<VoiceWorkDetailCollection> GetVoiceWorkDetailsAsync(string[] productIds, CancellationToken cancellationToken = default) public async Task<VoiceWorkDetailCollection> GetVoiceWorkDetailsAsync(string[] productIds, CancellationToken cancellationToken = default)
{ {
if (productIds.Length == 0) string[] validProductIds = [.. productIds.Where(x => !string.IsNullOrWhiteSpace(x)).Distinct()];
return [];
if (validProductIds.Length == 0)
string productIdCollection = string.Join(",", productIds.Where(x => !string.IsNullOrWhiteSpace(x)));
if (string.IsNullOrWhiteSpace(productIdCollection))
return []; return [];
string productIdCollection = string.Join(",", validProductIds);
string url = $"maniax/product/info/ajax?product_id={productIdCollection}&cdn_cache_min=1"; string url = $"maniax/product/info/ajax?product_id={productIdCollection}&cdn_cache_min=1";
var productInfoCollection = await GetJsonAsync<ProductInfoCollection>(url, cancellationToken); ProductInfoCollection productInfoCollection = await GetJsonAsync<ProductInfoCollection>(url, cancellationToken: cancellationToken);
return DLSiteToDomainMapper.Map(productInfoCollection); return DLSiteToDomainMapper.Map(productInfoCollection);
} }
public async Task<ReleasedWorksCollection> GetReleasedWorksAsync(ReleasedWorksRequest request, CancellationToken cancellationToken = default)
{
string locale = LocaleMapper.ToDLSiteLocale(request.Locale);
string date = request.Date.ToString("yyyy-MM-dd");
string url = $"maniax/new/work/api?locale={locale}&date={date}&period={request.Period}";
NewWorksApiResponse response = await GetJsonAsync<NewWorksApiResponse>(url, cancellationToken: cancellationToken);
if (response.Meta.Code != 200)
{
throw new InvalidOperationException(
$"DLsite returned code {response.Meta.Code}. " +
$"ErrorType: {response.Meta.ErrorType}. " +
$"ErrorMessage: {response.Meta.ErrorMessage}");
}
return DLSiteReleasedWorksMapper.Map(response);
}
} }

View File

@@ -1,8 +1,8 @@
using JSMR.Application.Integrations.Ports; //using JSMR.Application.Integrations.Ports;
using JSMR.Infrastructure.Integrations.DLSite.Serialization; //using JSMR.Infrastructure.Integrations.DLSite.Serialization;
using Microsoft.Extensions.DependencyInjection; //using Microsoft.Extensions.DependencyInjection;
namespace JSMR.Infrastructure.Integrations.DLSite; //namespace JSMR.Infrastructure.Integrations.DLSite;
public static class DLSiteClientRegistration public static class DLSiteClientRegistration
{ {

View File

@@ -0,0 +1,34 @@
using JSMR.Application.Integrations.DLSite.Models.ReleasedWorks;
using JSMR.Infrastructure.Integrations.DLSite.Models.NewWorks;
namespace JSMR.Infrastructure.Integrations.DLSite.Mapping;
public static class DLSiteReleasedWorksMapper
{
public static ReleasedWorksCollection Map(NewWorksApiResponse response)
{
ReleasedWorksCollection result = [];
if (response.Data.Products.Length == 0)
return result;
foreach (NewWorksApiProduct product in response.Data.Products)
{
result.Add(product.Id, Map(product));
}
return result;
}
private static ReleasedWork Map(NewWorksApiProduct product)
{
return new ReleasedWork
{
ProductId = product.Id,
Title = product.Name ?? string.Empty,
MaskedTitle = product.NameMasked ?? string.Empty,
Description = product.Description ?? string.Empty,
MaskedDescription = product.DescriptionMasked ?? string.Empty
};
}
}

View File

@@ -51,6 +51,7 @@ public static class DLSiteToDomainMapper
return new VoiceWorkDetails return new VoiceWorkDetails
{ {
Title = productInfo.WorkName,
Series = MapSeries(productInfo), Series = MapSeries(productInfo),
Translation = MapTranslation(productInfo, optionsSet), Translation = MapTranslation(productInfo, optionsSet),
WishlistCount = productInfo.WishlistCount, WishlistCount = productInfo.WishlistCount,

View File

@@ -0,0 +1,9 @@
using System.Text.Json.Serialization;
namespace JSMR.Infrastructure.Integrations.DLSite.Models.NewWorks;
public record NewWorksApiData
{
[JsonPropertyName("products")]
public required NewWorksApiProduct[] Products { get; init; }
}

View File

@@ -0,0 +1,15 @@
using System.Text.Json.Serialization;
namespace JSMR.Infrastructure.Integrations.DLSite.Models.NewWorks;
public record NewWorksApiMeta
{
[JsonPropertyName("code")]
public int Code { get; init; }
[JsonPropertyName("errorMessage")]
public string ErrorMessage { get; init; } = string.Empty;
[JsonPropertyName("errorType")]
public string ErrorType { get; init; } = string.Empty;
}

View File

@@ -0,0 +1,21 @@
using System.Text.Json.Serialization;
namespace JSMR.Infrastructure.Integrations.DLSite.Models.NewWorks;
public record NewWorksApiProduct
{
[JsonPropertyName("id")]
public required string Id { get; init; }
[JsonPropertyName("name")]
public string? Name { get; init; }
[JsonPropertyName("nameMasked")]
public string? NameMasked { get; init; }
[JsonPropertyName("description")]
public string? Description { get; init; }
[JsonPropertyName("descriptionMasked")]
public string? DescriptionMasked { get; init; }
}

View File

@@ -0,0 +1,12 @@
using System.Text.Json.Serialization;
namespace JSMR.Infrastructure.Integrations.DLSite.Models.NewWorks;
public record NewWorksApiResponse
{
[JsonPropertyName("meta")]
public required NewWorksApiMeta Meta { get; init; }
[JsonPropertyName("data")]
public required NewWorksApiData Data { get; init; }
}

View File

@@ -112,6 +112,9 @@ public class ProductInfo
[JsonPropertyName("title_name")] [JsonPropertyName("title_name")]
public string? TitleName { get; set; } public string? TitleName { get; set; }
[JsonPropertyName("title_name_masked")]
public string? TitleNameMasked { get; set; }
[JsonPropertyName("title_volumn")] [JsonPropertyName("title_volumn")]
public int? TitleVolumeNumber { get; set; } public int? TitleVolumeNumber { get; set; }
@@ -166,6 +169,9 @@ public class ProductInfo
[JsonPropertyName("work_name")] [JsonPropertyName("work_name")]
public string? WorkName { get; set; } public string? WorkName { get; set; }
[JsonPropertyName("work_name_masked")]
public string? WorkNameMasked { get; set; }
[JsonPropertyName("work_image")] [JsonPropertyName("work_image")]
public string? WorkImage { get; set; } public string? WorkImage { get; set; }

View File

@@ -6,8 +6,31 @@ namespace JSMR.Infrastructure.Integrations.DLSite.Serialization;
public sealed class DictionaryConverter<TKey, TValue> : JsonConverter<Dictionary<TKey, TValue>> where TKey : notnull public sealed class DictionaryConverter<TKey, TValue> : JsonConverter<Dictionary<TKey, TValue>> where TKey : notnull
{ {
public override Dictionary<TKey, TValue>? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) public override Dictionary<TKey, TValue>? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
=> JsonSerializer.Deserialize<Dictionary<TKey, TValue>>(ref reader, options); {
if (reader.TokenType == JsonTokenType.Null)
return null;
if (reader.TokenType == JsonTokenType.StartArray)
{
if (!reader.Read())
throw new JsonException("Unexpected end while reading array.");
if (reader.TokenType != JsonTokenType.EndArray)
throw new JsonException("Non-empty JSON array cannot be converted to Dictionary.");
return [];
}
if (reader.TokenType == JsonTokenType.StartObject)
{
return JsonSerializer.Deserialize<Dictionary<TKey, TValue>>(ref reader, options) ?? [];
}
throw new JsonException($"Unexpected token {reader.TokenType} when reading Dictionary.");
}
public override void Write(Utf8JsonWriter writer, Dictionary<TKey, TValue> value, JsonSerializerOptions options) public override void Write(Utf8JsonWriter writer, Dictionary<TKey, TValue> value, JsonSerializerOptions options)
=> JsonSerializer.Serialize(writer, value, options); {
JsonSerializer.Serialize(writer, (IDictionary<TKey, TValue>)value, options);
}
} }

View File

@@ -1,7 +1,7 @@
<Project Sdk="Microsoft.NET.Sdk"> <Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup> <PropertyGroup>
<TargetFramework>net9.0</TargetFramework> <TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings> <ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable> <Nullable>enable</Nullable>
</PropertyGroup> </PropertyGroup>
@@ -15,11 +15,16 @@
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
<PackageReference Include="BCrypt.Net-Next" Version="4.1.0" />
<PackageReference Include="HtmlAgilityPack" Version="1.12.4" /> <PackageReference Include="HtmlAgilityPack" Version="1.12.4" />
<PackageReference Include="Microsoft.EntityFrameworkCore" Version="9.0.10" /> <PackageReference Include="Microsoft.EntityFrameworkCore" Version="9.0.10" />
<PackageReference Include="Microsoft.EntityFrameworkCore.Relational" Version="9.0.10" /> <PackageReference Include="Microsoft.EntityFrameworkCore.Relational" Version="9.0.10" />
<PackageReference Include="Microsoft.Extensions.Caching.Abstractions" Version="9.0.10" /> <PackageReference Include="Microsoft.Extensions.Caching.Abstractions" Version="10.0.7" />
<PackageReference Include="Microsoft.Extensions.Http" Version="9.0.10" /> <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="10.0.7" />
<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="10.0.7" />
<PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" Version="10.0.7" />
<PackageReference Include="Microsoft.Extensions.Http" Version="10.0.7" />
<PackageReference Include="Microsoft.Extensions.Http.Resilience" Version="10.5.0" />
<PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="9.0.4" /> <PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="9.0.4" />
<PackageReference Include="NTextCat" Version="0.3.65" /> <PackageReference Include="NTextCat" Version="0.3.65" />
<PackageReference Include="Pomelo.EntityFrameworkCore.MySql" Version="9.0.0" /> <PackageReference Include="Pomelo.EntityFrameworkCore.MySql" Version="9.0.0" />
@@ -30,4 +35,8 @@
<ProjectReference Include="..\JSMR.Domain\JSMR.Domain.csproj" /> <ProjectReference Include="..\JSMR.Domain\JSMR.Domain.csproj" />
</ItemGroup> </ItemGroup>
<ItemGroup>
<Folder Include="Jobs\" />
</ItemGroup>
</Project> </Project>

View File

@@ -1,5 +1,6 @@
using JSMR.Application.Enums; using JSMR.Application.Enums;
using JSMR.Domain.ValueObjects; using JSMR.Domain.ValueObjects;
using JSMR.Infrastructure.Globalization;
namespace JSMR.Infrastructure.Scanning; namespace JSMR.Infrastructure.Scanning;
@@ -90,7 +91,10 @@ public sealed class DLSiteSearchFilterBuilder
public string BuildSearchQuery(int pageNumber, int pageSize) public string BuildSearchQuery(int pageNumber, int pageSize)
{ {
var (localeAbbreviation, localeCode) = LocaleMap.Map[_locale]; //string localeAbbreviation = LocaleMapper.ToAbbreviation(_locale);
//string localeCode = LocaleMapper.ToDLSiteLocale(_locale);
var (localeAbbreviation, localeCode) = LocaleMapper.Map[_locale];
using (var writer = new StringWriter()) using (var writer = new StringWriter())
{ {

View File

@@ -0,0 +1,62 @@
using JSMR.Application.Scanning.Contracts;
namespace JSMR.Infrastructure.Scanning.Extensions;
public static class DLSiteWorkExtensions
{
public static void InferAndUpdateExpectedDates(this DLSiteWork[] works)
{
// Precompute nearest known effective date on the left and right for each index.
var left = new DateOnly?[works.Length];
var right = new DateOnly?[works.Length];
DateOnly? last = null;
for (int i = 0; i < works.Length; i++)
{
var effective = GetEffectiveDate(works[i]);
if (effective.HasValue)
last = effective;
left[i] = last;
}
DateOnly? next = null;
for (int i = works.Length - 1; i >= 0; i--)
{
var effective = GetEffectiveDate(works[i]);
if (effective.HasValue)
next = effective;
right[i] = next;
}
// Fill only when BOTH sides exist and match.
for (int i = 0; i < works.Length; i++)
{
DLSiteWork work = works[i];
if (work.SalesDate.HasValue || work.ExpectedDate.HasValue)
continue;
DateOnly? previous = (i > 0) ? left[i - 1] : null;
DateOnly? nxt = (i < works.Length - 1) ? right[i + 1] : null;
if (previous.HasValue && nxt.HasValue && previous.Value == nxt.Value)
work.ExpectedDate = previous.Value;
}
}
private static DateOnly? GetEffectiveDate(DLSiteWork work)
{
if (work.ExpectedDate.HasValue)
return work.ExpectedDate.Value;
if (!work.SalesDate.HasValue)
return null;
// Bucket sales day to Early/Middle/Late => 1/11/21
var d = work.SalesDate.Value;
int day = d.Day >= 21 ? 21 : d.Day >= 11 ? 11 : 1;
return new DateOnly(d.Year, d.Month, day);
}
}

View File

@@ -1,16 +0,0 @@
using JSMR.Application.Enums;
namespace JSMR.Infrastructure.Scanning;
internal class LocaleMap
{
public static readonly IReadOnlyDictionary<Locale, (string Abbreviation, string Code)> Map =
new Dictionary<Locale, (string, string)>
{
{ Locale.Japanese, ("jp", "ja_JP") },
{ Locale.English, ("en", "en_US") },
{ Locale.ChineseSimplified, ("zh-cn", "zh_CN") },
{ Locale.ChineseTraditional, ("zh-tw", "zh_TW") },
{ Locale.Korean, ("ko", "ko_KR") },
};
}

View File

@@ -2,37 +2,26 @@
namespace JSMR.Infrastructure.Scanning.Models; namespace JSMR.Infrastructure.Scanning.Models;
public class DLSiteHtmlDocument public class DLSiteHtmlDocument(HtmlDocument document)
{ {
private readonly HtmlNodeCollection _workColumns; private readonly HtmlNodeCollection _workColumns = document.DocumentNode.SelectNodes("//dl[@class='work_1col']");
private readonly HtmlNodeCollection _workColumnRights; private readonly HtmlNodeCollection _workColumnRights = document.DocumentNode.SelectNodes("//td[contains(@class, 'work_1col_right')]");
private readonly HtmlNodeCollection _workThumbs; private readonly HtmlNodeCollection _workThumbs = document.DocumentNode.SelectNodes("//div[@class='work_thumb']");
public HtmlNode PageTotalNode { get; } = document.DocumentNode.SelectNodes("//div[@class='page_total']/strong")[0];
public HtmlNode PageTotalNode { get; } public DLSiteHtmlNode[] GetDLSiteNodes()
public DLSiteHtmlDocument(HtmlDocument document)
{ {
_workColumns = document.DocumentNode.SelectNodes("//dl[@class='work_1col']"); List<DLSiteHtmlNode> nodes = [];
//_workColumnRights = document.DocumentNode.SelectNodes("//td[@class='work_1col_right']");
_workColumnRights = document.DocumentNode.SelectNodes("//td[contains(@class, 'work_1col_right')]");
_workThumbs = document.DocumentNode.SelectNodes("//div[@class='work_thumb']");
PageTotalNode = document.DocumentNode.SelectNodes("//div[@class='page_total']/strong")[0];
}
public List<DLSiteHtmlNode> GetDLSiteNodes()
{
var nodes = new List<DLSiteHtmlNode>();
if (_workColumns.Count != _workColumnRights.Count || _workColumns.Count != _workThumbs.Count) if (_workColumns.Count != _workColumnRights.Count || _workColumns.Count != _workThumbs.Count)
throw new Exception("Work column node counts do not match!"); throw new Exception("Work column node counts do not match!");
for (int i = 0; i < _workColumns.Count; i++) for (int i = 0; i < _workColumns.Count; i++)
{ {
var node = new DLSiteHtmlNode(_workColumns[i], _workColumnRights[i], _workThumbs[i]); DLSiteHtmlNode node = new(_workColumns[i], _workColumnRights[i], _workThumbs[i]);
nodes.Add(node); nodes.Add(node);
} }
return nodes; return [.. nodes];
} }
} }

View File

@@ -19,7 +19,8 @@ public class DLSiteHtmlNode
public HtmlNode? SalesDateNode { get; private set; } public HtmlNode? SalesDateNode { get; private set; }
public HtmlNode DownloadsNode { get; private set; } public HtmlNode DownloadsNode { get; private set; }
public HtmlNode? StarRatingNode { get; private set; } public HtmlNode? StarRatingNode { get; private set; }
public HtmlNode ImageNode { get; private set; } public HtmlNode? ImageNode { get; private set; }
public HtmlNode? ThumbWithNgFilterBlockNode { get; private set; }
public HtmlNode[] GenreNodes { get; private set; } public HtmlNode[] GenreNodes { get; private set; }
public HtmlNode[] SearchTagNodes { get; private set; } public HtmlNode[] SearchTagNodes { get; private set; }
public HtmlNode[] CreatorNodes { get; private set; } public HtmlNode[] CreatorNodes { get; private set; }
@@ -55,7 +56,8 @@ public class DLSiteHtmlNode
//InitializeSalesAndDownloadsNodes(); //InitializeSalesAndDownloadsNodes();
StarRatingNode = GetStarRatingNode(); StarRatingNode = GetStarRatingNode();
ImageNode = GetImageNode(); ImageNode = TryGetImageNode();
ThumbWithNgFilterBlockNode = ThumbNode.SelectSingleNode(".//thumb-with-ng-filter-block");
} }
private HtmlNode[] GetGenreNodes() private HtmlNode[] GetGenreNodes()
@@ -165,10 +167,13 @@ public class DLSiteHtmlNode
// } // }
//} //}
private HtmlNode GetImageNode() private HtmlNode? TryGetImageNode()
{ {
HtmlNode linkNode = ThumbNode.SelectNodes(".//a")[0]; HtmlNode? linkNode = ThumbNode.SelectSingleNode(".//a");
return linkNode.SelectNodes(".//img")[0]; if (linkNode is null)
return null;
return linkNode.SelectSingleNode(".//img");
} }
} }

View File

@@ -0,0 +1,93 @@
using JSMR.Application.Enums;
using JSMR.Application.Integrations.DLSite.Models.ReleasedWorks;
using JSMR.Application.Integrations.DLSite.Ports;
using JSMR.Application.Scanning.Contracts;
using JSMR.Application.Scanning.Ports;
namespace JSMR.Infrastructure.Scanning;
public class ReleasedWorksProvider(IDLSiteClient dlsiteClient) : IReleasedWorksProvider
{
private const int MaxPeriodDays = 60;
public async Task<ReleasedWorksCollection> GetReleasedWorksAsync(VoiceWorkScanResult scanResult, CancellationToken cancellationToken)
{
DateOnly[] salesDates =
[
.. scanResult.Works
.Where(x => x.SalesDate.HasValue)
.Select(x => x.SalesDate!.Value)
];
if (salesDates.Length == 0)
return [];
HashSet<string> productIds = [.. scanResult.Works.Select(x => x.ProductId)];
DateOnly minDate = salesDates.Min();
DateOnly maxDate = salesDates.Max();
ReleasedWorksCollection collection = [];
DateOnly chunkStart = minDate;
while (chunkStart <= maxDate)
{
int endDayNumber = Math.Min(chunkStart.DayNumber + MaxPeriodDays - 1, maxDate.DayNumber);
DateOnly chunkEnd = DateOnly.FromDayNumber(endDayNumber);
int period = chunkEnd.DayNumber - chunkStart.DayNumber + 1;
ReleasedWorksRequest request = new(
Locale: Locale.English,
Date: chunkEnd,
Period: period);
ReleasedWorksCollection chunk = await dlsiteClient.GetReleasedWorksAsync(request, cancellationToken);
foreach (string productId in chunk.Keys)
{
if (productIds.Contains(productId) == false)
continue;
if (collection.ContainsKey(productId))
continue;
collection.Add(productId, chunk[productId]);
}
chunkStart = chunkEnd.AddDays(1);
}
return collection;
}
//public async Task<ReleasedWorksCollection> GetReleasedWorksAsync(VoiceWorkScanResult scanResult, CancellationToken cancellationToken)
//{
// DateOnly[] salesDates =
// [
// .. scanResult.Works
// .Where(x => x.SalesDate.HasValue)
// .Select(x => x.SalesDate!.Value)
// ];
// if (salesDates.Length == 0)
// return [];
// DateOnly minDate = salesDates.Min();
// DateOnly maxDate = salesDates.Max();
// DateOnly requestDate = minDate.AddDays(-1);
// DateOnly requestEndDate = maxDate.AddDays(1);
// int period = (requestEndDate.DayNumber - requestDate.DayNumber) + 1;
// ReleasedWorksRequest releasedWorksRequest = new(
// Locale: Locale.English,
// Date: requestEndDate,
// Period: period
// );
// return await dlsiteClient.GetReleasedWorksAsync(releasedWorksRequest, cancellationToken);
//}
}

View File

@@ -1,4 +1,5 @@
using HtmlAgilityPack; using HtmlAgilityPack;
using System.Text.Json;
using System.Web; using System.Web;
namespace JSMR.Infrastructure.Scanning; namespace JSMR.Infrastructure.Scanning;
@@ -45,4 +46,26 @@ public static class ScannerUtilities
return imageSource; return imageSource;
} }
public static string[] ParseJavaScriptArray(string value)
{
try
{
string json = NormalizeJavaScriptArray(value);
return JsonSerializer.Deserialize<string[]>(json) ?? [];
}
catch
{
return [.. value
.Trim('[', ']')
.Split(',', StringSplitOptions.RemoveEmptyEntries)
.Select(x => x.Trim().Trim('\'', '"'))];
}
}
private static string NormalizeJavaScriptArray(string input)
{
return input.Trim().Replace('\'', '"');
}
} }

View File

@@ -0,0 +1,94 @@
using JSMR.Application.Integrations.Chobit.Models;
using JSMR.Application.Integrations.Chobit.Ports;
using JSMR.Application.Integrations.DLSite.Models;
using JSMR.Application.Integrations.DLSite.Models.ReleasedWorks;
using JSMR.Application.Integrations.DLSite.Ports;
using JSMR.Application.Scanning.Contracts;
using JSMR.Application.Scanning.Ports;
using JSMR.Domain.Enums;
using JSMR.Infrastructure.Common.Languages;
namespace JSMR.Infrastructure.Scanning;
public class VoiceWorkIngestBuilder(
IDLSiteClient dlsiteClient,
IChobitClient chobitClient,
IReleasedWorksProvider releasedWorksProvider,
ILanguageIdentifier languageIdentifier) : IVoiceWorkIngestBuilder
{
public async Task<VoiceWorkIngest[]> BuildAsync(VoiceWorkScanResult scanResult, CancellationToken cancellationToken)
{
string[] productIds = [.. scanResult.Works.Where(x => !string.IsNullOrWhiteSpace(x.ProductId)).Select(x => x.ProductId!)];
Task<VoiceWorkDetailCollection> detailsTask = dlsiteClient.GetVoiceWorkDetailsAsync(productIds, cancellationToken);
Task<ChobitResultCollection> chobitTask = chobitClient.GetSampleInfoAsync(productIds, cancellationToken);
Task<ReleasedWorksCollection> releasedTask = releasedWorksProvider.GetReleasedWorksAsync(scanResult, cancellationToken);
await Task.WhenAll(detailsTask, chobitTask, releasedTask);
VoiceWorkDetailCollection voiceWorkDetails = await detailsTask;
ChobitResultCollection chobitResults = await chobitTask;
ReleasedWorksCollection releasedWorkCollection = await releasedTask;
List<VoiceWorkIngest> ingests = [];
foreach (DLSiteWork work in scanResult.Works)
{
voiceWorkDetails.TryGetValue(work.ProductId!, out VoiceWorkDetails? details);
chobitResults.TryGetValue(work.ProductId, out ChobitResult? chobit);
releasedWorkCollection.TryGetValue(work.ProductId, out ReleasedWork? releasedWork);
VoiceWorkIngest ingest = new()
{
MakerId = work.MakerId,
MakerName = work.Maker,
ProductId = work.ProductId,
Title = details?.Title ?? work.ProductName,
Description = work.Description ?? string.Empty,
Tags = work.Tags,
Creators = work.Creators,
WishlistCount = details?.WishlistCount ?? 0,
Downloads = Math.Max(work.Downloads, details?.DownloadCount ?? 0),
HasTrial = work.HasTrial || (details?.HasTrial ?? false),
HasChobit = chobit?.Count > 0,
StarRating = work.StarRating,
Votes = work.Votes,
AgeRating = details?.AgeRating ?? work.AgeRating,
HasImage = !string.IsNullOrEmpty(work.ImageUrl) && !work.ImageUrl.Contains("no_img", StringComparison.OrdinalIgnoreCase),
SupportedLanguages = details?.SupportedLanguages ?? [],
ExpectedDate = work.ExpectedDate,
SalesDate = work.SalesDate,
RegistrationDate = details?.RegistrationDate,
AI = details?.AI ?? AIGeneration.None,
Series = details?.Series,
Translation = details?.Translation,
Localizations = GetLocalizationIngests(releasedWork)
};
ingests.Add(ingest);
}
return [.. ingests];
}
private VoiceWorkLocalizationIngest[] GetLocalizationIngests(ReleasedWork? releasedWork)
{
if (releasedWork is null)
return [];
Language titleLanguage = languageIdentifier.GetLanguage(releasedWork.Title);
Language descriptionLanguage = languageIdentifier.GetLanguage(releasedWork.Description);
if (titleLanguage is not Language.English && descriptionLanguage is not Language.English)
return [];
VoiceWorkLocalizationIngest localizationIngest = new()
{
Title = releasedWork.Title,
Description = releasedWork.Description,
Language = Language.English
};
return [localizationIngest];
}
}

Some files were not shown because too many files have changed in this diff Show More